Shared publicly  - 
 
A lesson in shortcuts.

Long ago, as the design of the Unix file system was being worked out, the entries . and .. appeared, to make navigation easier. I'm not sure but I believe .. went in during the Version 2 rewrite, when the file system became hierarchical (it had a very different structure early on).  When one typed ls, however, these files appeared, so either Ken or Dennis added a simple test to the program. It was in assembler then, but the code in question was equivalent to something like this:
   if (name[0] == '.') continue;
This statement was a little shorter than what it should have been, which is
   if (strcmp(name, ".") == 0 || strcmp(name, "..") == 0) continue;
but hey, it was easy.

Two things resulted.

First, a bad precedent was set. A lot of other lazy programmers introduced bugs by making the same simplification. Actual files beginning with periods are often skipped when they should be counted.

Second, and much worse, the idea of a "hidden" or "dot" file was created. As a consequence, more lazy programmers started dropping files into everyone's home directory. I don't have all that much stuff installed on the machine I'm using to type this, but my home directory has about a hundred dot files and I don't even know what most of them are or whether they're still needed. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.

I'm pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.

How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about  40 years ago?

Keep that in mind next time you want to cut a corner in your code.

(For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)
2384
1108
roger peppe's profile photoPham Khuong Toan's profile photoHinsel Scott's profile photoking salman khan king's profile photo
111 comments
 
I don't think that hidden files, dot or otherwise, are inherently a bad thing to have. They're useful for storing metadata, e.g. .htaccess.

I do agree that they're far overused, and that using them for configuration that is not something directly and exclusively related to the directory that they're in is a bad and annoying practice, but I'd argue that the laziness and poor design was mainly in the failure to create a common and consistent place in the filesystem for per-user configuration, not in the accidental creation of hidden files themselves.

Interestingly, the failure to properly organize automated per-user data has repeated itself multiple times. Witness the horrible mess that is My Documents (and its later-introduced and rarely-used My X cousins) on Windows.
 
I bet some zealot will come here cry that it is now part of UNIX and, as such, must be a) adored as something from God itself and b) must be accepted as the best possible design for that feature. You're lucky for being there in the days and able to come and say how the things happened. Thanks :)
 
I was with you until you pulled the Plan 9 card, this is the OS with a kernel which tests for greater than or less than 0 to determine if an address cast to an int is in user space or kernel space.  It took me three weeks to get rid of that so I could get userland programs larger than 2gig on the ia32.  Talk about a hideous and short-sighted shortcut.  This is the OS with a shell (rc) which doesn't consistently match parentheses, taking away a valuable means of syntax checking in your editor.

Actually having a class of files be hidden is very convenient, we can place files which are executed on entry to a directory in the directory without the distraction of seeing them in ls's default output.  We can change our editor or shell's behavior based on where we are and presumably what we're doing, for example emacs and vi can behave differently when I'm in a directory which contains C source as opposed to one which contains roff source regardless of the file name's extension (there's you an abomination if you want a rant).
 
But without this concept, DecWars wouldn't have worked. How would Con Solo have hidden them all from the Administrative Flunkies???
 
I've always thought the littered ~/.foorc problem would be more likely do to the whole portability hell and who can really know if you're gonna want to port stuff to an OS without directories.

That and people are two lazy to write ~/.cfg/foo or ~/.rc/foo
 
Sadly changing KZERO was only a tiny part of the solution, there were over 700 places where assumptions were made regarding the MSB of an address, every one of these required scrutiny and change.

I suggested above that having a class of files which is not visible by default is a good thing, whether it be done by placing a "." at the front or by creating a filesystem attribute is immaterial (to me).  Most programs allow you to change the config file name on the command line so it is probably not nearly as difficult as you might fear.
 
It seems that MS-DOS learned and fixed the mistake :). One of the shortcuts I despise most in the "C" language are the strstr() and stristr() - this is the most abused function I've seen in the code I have to work on. So bad, that it probably creates it's own weird worlds with strange laws and mysterious overlords.

As an example, OpenGL's extensions are often checked with strstr() - they are only delimited by spaces in one big string.
 
I really don't like the *nix approach of a "hidden" file. For me, Windows' approach is better. In *nix you actually have to change the file name to hide a file. In Windows, this is not the case; the file's name stays as it is.
 
basking in the geeky goodness ohmmmmmm...
 
But in Windows (NTFS) you have streams, that are even more strange, yet useful sometimes.
 
What was the filesystem like prior to hierarchies?
 
I'm so glad to know the story from you. Excellent post, thank you!
 
+Dimiter Stanev Whats your point? Linux has those too, and has had them for decades, they are just called FIFO.

I agree, in fact, if i see code that will produce these type of issues at work, I always question the developer why its there in the first place.

Although some developers may tell you otherwise, bugs are not features.

It would be interesting to look at the POSIX standard to see if this was actually an intended feature or simply a bug.
 
To me dot files are useful and are good for saving personal configurations, while I agree that they should should be in one central location in the user's home folder.
 
As the post points out, a great way to hide files from enumeration is to prefix them with "cfg/". Magic prefixes to file names and extra bit attributes just add complexity in this case. 
 
It was a little more complex than you suggest as many of the early systems were not big enough to handle Unix. Memory was very limited and dot files were a means of breaking up an over-large program into loadable sub-sets that the 16k of RAM could manage. As RAM expanded so the systems evolved but as equipment costs were high many programs had to work on older machines to be profitable.
 
+brendan edmonds What +Dimiter Stanev is talking about is not FIFOs or even SYSV Streams.  NTFS allows files to contain more than one stream of bytes, each of which have an independent range of file positions starting at zero.

Microsoft did this as a generalization of the original MacOS concept of a file having two forks, a data fork and a resource fork.  I'd claim that it was a mistake on the Mac, and a bigger mistake for Microsoft to build it into NTFS.

Certainly POSIX has nothing to say about it, and there is no POSIX-compliant way to access the alternate streams of an NTFS file.

Also, what POSIX has to say about a Unix feature is in no way indicative of whether that Unix feature was originally intentional or accidental. POSIX started to codify existing practices, though they went beyond that in some areas.
 
+brendan edmonds Heard about FIFO's, but didn't know they are used as metadata. Don't know much about them, which means I'll spend some time reading :)
 
Wait a minute!
So, using the dot prefix to denote hidden files was not a good idea.
And how about using upper-case/lower-case in Go language to denote public/private symbols?
 
+Denilson Sá .dot files came as side effect of a bit of cowboy coding, but the benefits are great (I like .dotfiles - they have this geek-ish character). Use of upper/lower case to distinguish external/internal symbols was conscious decision, not something that just happened. I guess that's not a very great argument, but it shows the intent up front, not - "Hmm.. that's very cool, now that it happened"
 
+Eric Smith So your saying NTFS has a more complicated version of FIFOs?, kind of like UNIX sockets? 

Or is more like a group of FIFOs in one file?

Can you provide me a link about how NTFS uses streams?
 
Wow, I had no idea. I always thought of that as a 'feature', I feel that I've almost have been indoctrinated into believing that all your config files are in some 'dot' directory on your home dir and that it is a good thing for people not to see them. I'm also not sure that it is either bad or good.... :( i've even alias ls='ls -al' on my shell for that reason. Feeling like i've missed something for a long time.... I wonder what else i've missed. 
 
Love it, Rob.  You may be ornery and  a bit of a zealot (still pushing
ed(1) on your friends at work?), but you never fail to educate and
entertain when you focus your laser-like attention on something.
 BTW, not only have I confirmed that "cat -v" is in fact Harmful, but
turns out it's also the root cause of Global Warming -- who knew? 
Bad BSD!  Shame on us  ;) 
But politically speaking (i.e. Passing the Buck) if you look at where
our funding came from, it really was the War Departments fault -- oops, I mean the DoD's Darpa: purveyor of peace through surgical first
strikes...

-- Marc Teitelbaum
 
Now I'm wondering what the "very different structure" was that the filesystem had before Version 2.
 
We have never met, but I have used plan9 in a vm. I'm no big wig, but I've always been curious about how these thing work. Too bad bell labs got bought out by att. I've noticed that many servers on the att network still maintain the BellSouth name, at least in these parts. Do ya'll plan on developing it plan9 further? I started off using windows, so hidden files were nothing new to me when I started using Linux. I thought having to use an escape character for cding to a directory was kinda lame. Anyway, good luck.
 
Changing permissions to remove read priveledges from the user does essentially the same thing as hiding the file, the user can no longer see it, you have to be a super user to see it.

I read somewhere that Ctrl+Alt+Del was a shortcut hack put in by MS dev's while developing early iterations of Windows.  It was only after beta testers got used to it that it stuck, and so its still there today, because some dev wanted a quick and dirty way to reboot the system.

UNIX / Linux operating systems are full of some "unusual" commands.  For instance, most commands follow the same general syntax, but then there's odd ducks like dd which has the honor of carrying the nickname "data destroyer" for it's ability to erase or overwrite large quantities of data with a simple mistake.

I always find it odd when reading man pages, and stumbling across how to do things like list a process and all of it's "dead children" or "zombie processes" in the top man page.

Meh, it's history, it's part of the software and the culture now, love it, hate it, it is what it is and I don't believe it can be reversed.
Rob Pike
+
3
4
5
4
 
dd is horrible on purpose. It's a joke about OS/360 JCL. But today it's an internationally standardized joke. I guess that says it all.
 
If an application places 100k files in a single directory, do you blame the OS or the application author?

Just cause you can create dot files doesn't mean everyone and their dog and cat should be filling your home directory with dot files.
 
we could also argue that the whole file system is a big set of pointers. so symbolizing "where I am" with a dot makes sense (the dot means, "Hey you're here". In the same logic, dotdot means where you came from (your parent folder).
Long story short, you're right, it created a mess as well as a feature.
 
+Paul Michael Bauer It seems to me like an overly complicated way of hiding data from the user. In fact it is used by most Viruses to hide its code from anti-virus applications.

If you really need to use this feature, simply change the ":" to a "." (i.e fifo:stream1 to fifo.stream1 and fifo:stream2 to fifo.stream2), Why no just use different files like this is a beyond me.
 
+Dan O'Connell Ctrl-Alt-Del was a BIOS command on the IBM PC long before Windows was an idea. When PCs were single-user being able to soft-reboot was a pretty important thing.
 
Wait, the default file manager on Android still hides these "dot" files
John H
+
2
3
2
 
Replace cfg/ with '.'? I like it - migrate all those snotty cfg users to the unix way >;-D
John H
+
1
2
1
 
Last comment was in response to Russell Nelson, sorry.
Russ Cox
+
2
0
1
0
 
+Michael Galassi I'm not sure what version of Plan 9 you are talking about. Splitting the address space in half between user and kernel was completely reasonable for the original PC port, which was to a 386 laptop with something like 8 MB of physical memory, in June 1991. Linux and FreeBSD didn't exist yet, but as they did come into existence, they both made the same reasonable decision. You have to pick a line, and there are good technical reasons to pick 2 GB. Even today, x86-64 kernels split the 64-bit address space in half the same way.

I changed Plan 9 a few years ago so it could run 3.5+ GB user processes on x86, and it was really not a big deal. It did not take anywhere near three weeks. It might have taken a day or two. And it's in the standard distribution.
 
Thats awesome! Love those historical geeky tidbits of knowledge! As a bonus I bet I score a few extra points at bar trivia next week! ;)
 
I messed up my quote! I'll do it again properly: "This is a UNIX system, I KNOW THIS!" cue velociraptor scream 
 
Fascinating. Personally, I use aliases/shell functions that invoke ls with the "-A" option, so "hidden" files aren't really hidden.
 
+Rob Pike, why do so many (old school) CS people say "assembler" when referring to assembly code?
 
+brendan edmonds About streams in NTFS - Think of them as attached files to a file. If a file has streams, and I move/copy/delete this file - the streams go along with the file (well for copy this is true only for the Win32 CopyFile, or shell's copy - otherwise you have to copy one by one)
Cong Ma
+
5
6
5
 
This is how evolution works.  Mutations happen, and the fittest survive.  It may have originated as an ignoble "bug", but so did all the genetic mutations that are responsible for whatever intelligence we humans proudly possess.  I agree that the dotfile convension may not be strictly necessary and it indeed sees frequent abuse.  Ideally developers should be civil with respect to the users $HOME, document the creation, location and purpose of dotfiles, offer means to migrate dotfiles across incompatible upgrades, and clean them up on uninstall.  But then again if you ask for that you don't understand coders. ;)
 
.git is my favorite use of dotfiles
 
There seems to be a mistake in thinking that "dot" files are limited to per-user config. There are plenty of cases where they are used for per-directory config. For example, .git, .htaccess, .project, etc. The reusable convention we have is ever so useful, allowing us in any location to separate those files we always want to see with ls from those files we only sometimes want to see with ls -a.  Far more reusable than a restrictive convention such as a $HOME/cfg directory.  If it were not for "dot" files, we would still have "lazy programmers" only there'd be an increased chance that if we type ls -F anywhere we'd get:     

thisConfig/ thatConfig/ theOtherConfig/ DStoreThumbnails/ UnimportantConfig/ BuryingConfig/ TheConfig/ FilesConfig/ YouConfig/ ReallyConfig/ CareConfig/ AboutConfig/ InConfig/ AConfig/ HopelessConfig/ BarrageConfig/ OfConfig/ SlightlyConfig/ DifferentlyConfig/ NamedConfig/ ConfigConfig/ DirectoriesConfig/ superimportantfile AllConfig/ GettingConfig/ InTheConfig/ WayConfig/ EvenConfig/ IfConfig/ YouDontConfig/ UseConfig/ AMinusAFlagConfig/
 
I find extra amusement in this because I once wiped out a lot of users' data with a command I had written to clean up dot files in a test home dir. Of course, it not only matched '.' but also '..' and happily ate half of the other home dirs on the system. I was fortunate to have just made a full backup.
 
William: .htaccess files are just plain bad. Having to stat() the file system recursively for every fresh http request is not really optimal. One should always use $httpd.conf for configuration.

And we can already a shift towards (almost) one centralized configuration directory for every user with XDG using .config 
 
I detest dot files. It's annoying when you plug a usb into a Mac and get loaded with them for no reason.
 
Thanks! Always wondered how those dot files happened.

Of possible interest: MaxOs X has done kinda-sorta some of these suggestions.

They went from forks and avoided ntfs style streams to this radical idea called a directory ;-).

Directories named 'Library' , which in macland are usually config directories, are now hidden from GUI finder (but not ls).


 
Now that the (Linux) revolution is moving everything below /usr, a big step could be made by having /usr/usr/$user and /usr/etc/$user.

But then you said that the directory hierarchy »appeared, to make navigation easier.« Hopefully people at Red Hat are reading what you wrote.
 
I dont think it would be hard to fix this, even if you had a directory called CFG which would contain all the `.` files in that would be in parent directory.  Not ideal but would be exatly what you have in plan 9.

That said users should treat there $home directory as a root directory and not clutter it up when they can have sub directory's for the tasks at hand.
 
i've found the dot configuration folders to be pretty convenient, when i've needed to go tinkering.  maybe i'm just used to them, therefore biased towards them
 
TOPS20 User's manual :
6.14  VISIBLE AND INVISIBLE FILES
 
Typically, you will have files that  are  not  currently  in  use  and cluttering your directory.  You can clean up a directory by moving the files to tape (archiving), or to other directories, or by making  your infrequently used files invisible.
 
An invisible file is not displayed by a simple DIRECTORY  command  and is  not accessible to programs and EXEC commands.  The ARCHIVE command automatically makes  files  invisible.   When  you  RETRIEVE  archived files, they will remain invisible when restored to disk.
 
To make a file invisible, use the SET FILE INVISIBLE command.  To make an invisible file visible again, use the SET FILE VISIBLE command.  To display your invisible files,  use  the  DIRECTORY  command  with  the INVISIBLE subcommand.
 
 
Just remembered my early GCOS programming days.  There user login shell had a ..init file which was executed upon startup.  `.` files been around a while it seems on many non nix platforms.
 
I have no bloody idea what you blokes are talking about. But the next time I want to sound important, when instructing staff how to do their job. I'll be telling this story and finishing with "... And that is why lazy people get fired"
 
unix and windows file systems should have not been hierarchical in the first place. that is the reason why they are slow and inneficient. they should have been flat with built in metadata for them to be more efficiently searched. but gee.. its hard to undo it now because everthing is based on it.
 
The filesystem might be faster, but the limitations of storage hardware would have made metadata querying even slower. 
Translate
 
+Jonathan Brown I don't remember any system where program overlay files used a "." prefix? Which OS are you thinking of?
 
Those were the days of CP/M and Dos. When if 1k actually equalled 1,000 most systems would not work because of insufficient memory. The majority of a programmers time was devoted to making programs as efficient as possible. Where peek and poke had very different meanings.
 
I guess the trend these days is to "pollute" $HOME/.config instead of $HOME itself. In that sense, Mr. Pike, seems like we're making progress, but it's not 100% yet. I still think fs objs not shown by default has some merit; the MS family of OSes/fs implement this with the H attribute bit for example.
 
That makes sense. So files would have flags or labels instead of being placed somewhere in a directory structure. Was that how Unix was before the hierarchical system was used?
 
I was snoring during all of this. My two cents is not every config
file needs super user privaledges and not every program in today's
Linux distros and UNIX environment are necessary. I mainly use wmaker
or fluxbox and experiment with what makes Linux distros tick. What can
I remove without sacrificing its security or durability. Anyway, I'm
going back to bed, not even daylight and ya'll are chattalattin like
crazy.
 
+Michael Galassi 
 "This is the OS with a shell (rc) which doesn't consistently match parentheses"

Really? As an rc user I'm a little surprised by this assertion.
 
file systems are outdated. they are slow. use SQL^WNoSQL databases with XML BLOBs!
</parody>
 
"][" {()} {()} ][_,  pretty cool heheh
 
Personally, I tend to agree with the article. I'm unsure as to whether it is the real origin, however, this is also a terrible standard from a user perspective. If a user renames a file with a . in front, they shouldn't expect it to simply disappear.

Windows does this properly, with a specific attribute. Users can hide files as they wish, without the risk of breaking anything. And it isn't any easier to type mv abc .abc than hide abc; unhide abc (in fact, the latter is actually easier) 
 
"bugs" quite often become "features" when their usability outweighs their consequences...
 
+Rob Pike dd is awesome guys. Especially when piped to (nc) for forensic cloning over IP. Just because it's capable of ruining your day if you typo doesn't mean it's bad programming. We're you looking for an "are you sure?" question? Use windows then.
 
While I agree in principle with your arguments tat shortcuts are nearly always a bad idea, from your example its not always the case tat it was as simple as you say either.

Quite often when code is being built up code that has been previously written may seem like its producer may have been taking shortcut, however in practice what usually happens is that the code created originally for one purpose is then used in a way that it was not first intended.

Its these extra use cases that make it seem like a shortcut. When this happens the code should be re-factored to include the new use case.

Good code may look like rubbish with the advantage of hindsight.

The real mistake was not refactoring.
 
The history of software is rife with such issues. Little things, but they've added up over time to become quite substantial. What I'd love is a way to go back in time and fix these things properly (but not end up with a worse concept like 'the registry').

Paul.
 
The new XDG standard uses the ~/.config directory for configuration files. Quoting from http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html :

The XDG Base Directory Specification is based on the following concepts:
 - There is a single base directory relative to which user-specific data files should be written. This directory is defined by the environment variable $XDG_DATA_HOME.
 - There is a single base directory relative to which user-specific configuration files should be written. This directory is defined by the environment variable $XDG_CONFIG_HOME.
 - There is a set of preference ordered base directories relative to which data files should be searched. This set of directories is defined by the environment variable $XDG_DATA_DIRS.
 
yessssssss ilike ping-pong !
Translate
 
and yet it's a good "glitch" if you want to call it that way. the alternative is what? windows registries? when you want to migrate linux systems you migrate dotfiles and your /etc and that's it. when you do migrations on windows you do what?

we can version them in git and easily share them. that xdg base crap is just really annoying. and then we have gnome and kde databases on top of them.

what's the result? suddenly the software layer itself has to take care of the backups instead introducing an extra layer of faults and obfuscation where there shouldn't be.

but yes i guess metadata in .cache according to xdg specs makes sense
 
I was wondering why this post had so many comments. Then I found out it was posted on proggit/HN. The discussions there were hardly interesting, however.
 
$HOME/cfg and $HOME/lib are a terrible idea. They prevent you having two differently-configured instances of the same object within the same hierarchy (e.g. both owned by the same user). Actually I will go further, any absolute path that has special meaning makes it difficult to create two subtly-different instances of the same thing.

How does Rob test an updated library without breaking the current instance? I hope he doesn't put the new version in $HOME/lib.

Or to put it more succinctly, the '.' prefix allows me to have two or more instances of the same thing with the same name (by varying the parent folder). Absolute paths don't allow this.
 
+Mark Lomas Are you familiar with Plan 9's bind(1)?  It would solve the problems you posit.
 
+Mark Lomas The . prefix doesn't do that, having a convention for a local configuration file does that. You can use a different convention.
 
there is no feature so bad that some people won't love it.
i could post examples but i don't want to cause any more flaming in this thread.
Rob Pike
+
2
7
8
7
 
Turning off comments.