Profile cover photo
Profile photo
Warren Young
60 followers -
Programmer. Photographer. Audio geek. Renaissance human.
Programmer. Photographer. Audio geek. Renaissance human.

60 followers
About
Warren's interests
View all
Warren's posts

Post has attachment
I Am a Software Developer

I stopped using the term "software engineer" to describe what I do for a living about a decade ago.

To me, the line dividing “engineering” from other applied technical tasks is simple: engineering is applied physics. If you’re doing something that doesn’t have an underlying physical equation — or there is an equation but you haven’t got the training or skill to produce that equation on demand — you aren’t doing engineering.

Software development is not engineering, not because some governing body says so, but because most software is so far removed from physics that it has also gone beyond any pretense of engineering.

I am not arguing against professionalism, or doing the right things right. I’m also not a linguistic prescriptionist (http://goo.gl/ntxDpO) attempting to hold back the normal evolution of language. I hold this line because it is a useful dividing point, and I don’t want to see the line blurred to the point of uselessness.

I came to this realization after, having spent most of my life learning software development skills, I taught myself electrical engineering. At some point during that journey, it became blindingly clear to me that the two disciplines are radically different in their underlying attitude toward robustness.

If I build an electrical circuit that oscillates when I turn it on, and I did not intend to build an oscillator, I have two major paths to fix it:

1. I can start whacking capacitors in semi-randomly to try and damp the oscillation out or modify the phase response of the system in a way that prevents the oscillation from starting.

2. I can do the circuit analysis that mathematically gives me my circuit’s poles and zeroes, construct a Bode plot (http://goo.gl/4fuWtA) and then see why my circuit is oscillating and what it will take to fix it.

The mere availability of option 2 is why electrical engineering is engineering. We simply don't have anything like option 2 in software development.

The nearest equivalent example I can come up with in software is a multithreaded software system which, on one out of every thousand runs, core dumps in a random location because it has $CATALOG_ITEM that’s caused by $RARE_CONDITION, where $CATALOG_ITEM is covered on one page of the half dozen books I have on multithreaded programming and its various traps. (http://goo.gl/r3Crhs) I can’t produce an equation or graph that demonstrates why my concurrent system has this problem, which means I don’t have a way to find the value of the software component I must insert to fix it, or where to insert it. I'm probably going to have to resort to a fix that resembles option 1, which is hacking, not engineering.

There are gray areas where software development does turn into engineering.

If you're writing software that models a physical phenomenon or uses equations of physics to solve real-world problems, then you're really just doing physics, which nulls my argument out, rather than disproving it.

Another very different case is when you’re writing software that can be understood using the same sort of math you'd use to do physics. There are software systems that can be understood through control theory (http://goo.gl/5kbLYu), for example, which means problems in those systems can be fixed using the same methods you would use to fix an electrical circuit with unwanted oscillations. You might not call the fix for the software system a capacitor, but the underlying principle is the same.

My distinction remains useful because very few practical software systems fall into one of these gray areas.

Those reading this who primarily do embedded software development may be having a hard time accepting this argument, but I believe that is because so much of what you do comes back to either electrical engineering or to an interaction with the real world, both of which amount to doing physics of some sort.

The distinction I’m interested in occurs when you build an abstract software system with no essential tie to physics. When we break those bonds with reality, we also break the closest bond our system has to physics, whether that be the electronics controlling the CPU chip, the optics in the input device, or the electro-magnetics in the storage system. That distance prevents me from using the physics equations governing those subsystems to reason about the behavior of the software system running on top of them.

That is to say, I can write an equation for what goes on inside my optical mouse, but that equation doesn't help me to understand why my web GUI layout jitters oddly when I move my mouse over a certain DOM element in IE while the same software works just fine in Firefox.

I brought up robustness above. I’ve studied a bunch of the efforts people have made to try and make software more robust. Some are more successful and practical than others, but none of them are anything like physics.

The best hope I have seen for a more robust software future is functional programming, which attempts to reduce software to pure mathematics, which is a) not the same thing as physics; and b) has severe limits, in that any part of an FP software system that causes side effects suddenly can’t be treated with the underlying mathematics of FP, and so must be walled off, to keep the pure side of your FP system pure. (Thus Haskell's famous I/O monad: http://goo.gl/7c1w8P)

That merely restates my argument’s premise: physics is all about side-effects. If I hit this cue ball with this stick at point p on the ball with velocity v, it will go in this direction at this speed, decreasing according to air drag and table friction. If it hits something, I can predict what will happen to the cue ball and the thing it hits, all without doing a practical experiment, because a bunch of smart guys figured out the physical equations for this kind of thing hundreds of years ago.

That’s what makes billiards a sport rather than a branch of experimental physics. Given all of the initial conditions, we know what the result will be; we don't have to try it and see. Billiards is a sport because we can’t entirely account for the human variable. No billiards fan has a favorite cue ball, showing up to matches that cue ball participates in, collecting trading cards depicting that cue ball and hoping to get its signature on that card one day… Each cue ball is more or less interchangeable with all other cue balls, because physics.

Meanwhile in software land, the closest things we have to interchangeable components are industry standards, and you all know how that story goes: https://xkcd.com/927/

I am a software developer, not because both Canada and the State of Texas refuse to recognize me as a software engineer, but because I almost never get a chance to solve my software problems with physics.

Kicking the web server doesn’t count as “doing physics.” :)


Post has attachment
Every now and then here in the US, you'll hear someone talking about drilling through the earth "to China." If you're like me, you took this without any thought, but on reflection, it actually makes no sense: China and the US are both in the Northern Hemisphere! Two antipodal points on the globe must be in opposite hemispheres.

It's one thing to read a fact like that in print. The thing that brought it home to me is a few minutes playing around with the Antipodes Map. It turns out that all of the lower 48 of the US is opposite...the Indian Ocean.

That's right, pretty much zero land is opposite the US on the globe.

It isn't completely zero. The northernmost parts of Alaska are antipodal to Antarctica, Hawaii is antipodal to southern Africa, and there are a few islands out there in the IO standing opposite the continental US. For example, standing opposite southeastern Colorado is a tiny island called Île Amsterdam, with a total population of about 20 scientists, according to Wikipedia.

Percentage-wise, I'd guess less than 1% of the total US landmass is antipodal to some other landmass.

Post has attachment
Why I Like the F# Language

James McCaffrey wrote a blog post explaining why he doesn't like F#, a language I do happen to like.

In fact, F# is my favorite functional programming language, hands down. I've gone out of my way to install Mono and Xamarin on machines running non-Microsoft OSes in order to use F# there, too. It's that good.

So, naturally when I see this piece dumping on one of my favorite programming languages, I read it. And...wow. You'd think a guy who proudly claims to have come up on K&K BASIC (mid-1960s!) would have a better sense of how to conduct a language advocacy argument. Someone with that much experience should have been through many MFL (My Favorite Language) battles and learned from that experience how to avoid the sorts of logical fallacies and weak arguments we find in this post.

McCaffrey starts off his blog post with two pieces of sample code in a screenshot. Please don't do this. Use <pre>, or Gist, or Pastebin, or something that lets us copy and paste the program text.

The blog post never comments on these two code samples in prose. I suspect many reading the post won't even click the screenshot to see what it shows, and those who do click are going to see an argument that amounts to "amirite?"

Well, I did click on it, and I'm here to tell you that no, you are not right, Dr. McCaffrey. These two pieces of code write the same text to the console, but they do not do the same thing.

The F# snippet he picks on is clearly showing off the power of F#'s first-class functions. Its primary purpose is not to be the simplest way to write a given series of strings to the console, but that's all his C# response does. A fair comparison would have required use of C#'s verbose delegate syntax to mimic F#'s first-class functions. (http://goo.gl/MLun14) I'm no C# expert, but I believe that would make the resulting C# several lines longer than it already is.

Given that I do not have the chops to do that fair port, I chose instead to rewrite McCaffrey's C# code in F#: http://goo.gl/g8POBl

If you visit that link, you will find an F# program with the same number of lines as the original F# code that McCaffrey started with, but with generally shorter lines. This is because my version of the program does the same actual work as the C# alternative. Depending on how you look at it, my F# code is anywhere from about 50% to 200% smaller than the C# code, and yet it's just as clear. Now it's a fair comparison.

This brings me to part of my answer to his complaint, "F# has no compelling technical advantage." In study after study, computer science researchers have found that the number of errors made per line of code is only a loose function of the language itself. This tells us that the number of errors per line of code is partially a function of the number of lines! Therefore, a language that lets us express a solution in fewer lines of code is likely to result in fewer errors.

This makes complete sense when you think about it. Fewer lines means you can put more code on the screen, which means you have more context in front of you while thinking about the problem. Fewer characters means fewer possibilities for typos. Fewer lines means you're probably leaving more about your program's functionality to the language runtime and libraries, which are presumably already well-debugged.

This is the same sort of argument that C# advocates routinely make to C++ programmers. It seems that Dr. McCaffrey has developed a blind spot when it comes to languages like F# that take the same attributes a step or two further. 

I'm taking McCaffrey's points out of order here, so let's get back on track.

His first formal point is that "F# just doesn’t feel right." Well, I'm here to tell you that it does feel right to me. So much so that when I'm looking at a bit of F# code, it is more obvious whether it is correct to me just from looking at it than with C# code.

I believe this is partly due to the compact nature of F#, which removes unnecessary noise from the code without turning it into a form of encryption. This is a difficult balance to strike. There are other languages that manage to be even more compact than F# by taking this too far, so that working out what a single line of code does might take minutes of study. Some people take the wrong lesson from that: verbosity is not a cure-all. There is a vast world between APL and COBOL; the perfect balance between compactness and clarity is somewhere between those extremes, not out at the ends. I think F# strikes an uncommonly nice balance here.

Another reason I believe that correct F# code is more obviously "correct" from simple examination is due to the language's mathematical rigor. F# simply disallows huge classes of dodgy code, so that you don't have to puzzle through those possibilities when studying a piece of code. This frees mental power up to study the code on its own terms, since you no longer need to second-guess what the compiler has actually done under the hood.

In my experience, F# code is more often correct as soon as it compiles as compared to code in less strict languages. This happens because the F# compiler is able to find many of the common sorts of bugs that other languages implicitly allow. I mention some of these features elsewhere in this post. There are others, which I won't discuss here: F# doesn't allow implicit "else" clauses in "if" statements, F# doesn't let you implicitly ignore return values, F# makes a strong distinction between variables and values, F# doesn't allow implicit conversion to or from "null," etc.

I suspect this is what McCaffrey means when he complains that F# seems to fight him rather than help him. My guess is that he's getting annoyed by F# when it refuses to allow code constructs that C# would tolerate.

I rarely find a case where an F# compiler error is a result of F# being obtuse or overly restrictive. I usually find that whenever F# is trying to prevent you from programming in a certain way, there is a good reason for it, and it's worth your time to learn a new coding habit. This is not a matter of kowtowing to the tool. Even when I am not actually writing F#, I frequently find myself choosing modes of expression inspired by F#. I believe these matters of style are objectively superior, worthy of emulation in any language where they have applicability.

McCaffrey's next complaint is that "F# has a tiny user base." He bases this on just two data points, one of them unsourced. That sounds like cherry-picking to me.

First, he posts numbers retrieved from some unnamed job board. Lacking a source for that data, have no way of evaluating whether it is representative of the software development job market as a whole. For all we know, this job board happens to be particularly popular among C# recruiters.

On top of that, he's ignoring the fact that many companies asking for C# talent are also implicitly asking for F# talent. The simple fact is that the PHBs often don't know what technologies they should be asking for. That's why they hire techies. For all management knows, half their "C#" code base is actually F#. How many times have you overheard someone in management above you make a technical claim about the system you are working on which is simply incorrect? How is it that we expect these same hiring managers to specify the skill sets they actually need when hiring? Second-guessing job descriptions is practically a sport in software development circles.

This sort of "data" also bugs me when it's used in other contexts, such as in OS advocacy arguments. (Mac OS X vs Linux, or Windows Mobile vs Android, or...) Someone will point to the size of an app store or package repository or download site as proof of a platform's superiority. Just as I don't need 1,000,000 apps for my phone, I don't need 30,000 job offers. I usually only need one of a kind at a given time.

There's something to be said for being uncommonly capable at serving a lucrative niche. If you're one of hundreds of thousands that know some very popular language, you are to some extent interchangeable. Your employer is spoiled for choice when it comes to your replacement.

I don't mean to imply that you should write code in obscure languages purely as a form of job security, but you might prefer to seek employment where they don't treat you like a machine cog. You should prefer to work at places that let you choose the right tool for the job, even if it's something the next guy up the reporting chain hasn't read about yet in PHB Weekly Magazine.

That said, a certain amount of conservatism in choice of tools is a good thing. It is usually a bad idea to choose the 101st most popular alternative from a list of applicable tools. But at the same time, I don't see a problem with choosing from the top 25 or so. As I write this, F# is at #11 on the TIOBE Index, and it's been rising. (http://goo.gl/1yyfu0) We can argue all day about the problems with TIOBE's methodology, but I think it is fair to infer from this that F# is probably not going to go away before your next project is complete. That's as good as a tool needs to be in this business.

Given that F# is now open source, it's probably not going away ever, in fact, in the same way that fountain pens and AM radio aren't going away. If you need it to be there 20 years from now, there will be a way to get it running again even if its key developers all inexplicably abandon it tomorrow.

The second data point McCaffrey uses to support his claim that the F# user base is "tiny" is even easier to knock down. In his position as editor at a magazine aimed at programmers, he sees the hit rates on the articles' web pages. He infers from the low hit rate of the F# articles that there just aren't many F# programmers. He's made a classic error in demographic statistics: self-selection bias. If your magazine builds a reputation as a C# journal for many, many years, then starts publishing a few F# articles, who exactly do you suppose is going to be choosing whether to read them? C# programmers, that's who! You might as well try to suss out the popularity of iOS by publishing an article about the Swift programming language in the same magazine.

Now we come back to that final point in the article, "F# has no compelling technical advantage." In addition to being shorter, as discussed above, F# code is usually also stricter, in a good way.

By way of illustration, let me also take on his stated preference for the way C# makes you be explicit about your data types. He's made a common mistake here, that type inference means you aren't telling the compiler what your data types are. This confuses dynamic typing with static type inference. They may appear to do the same thing when you're looking at code in an IDE, but under the hood, these two approaches to type systems do very different things. If you don't understand the distinction, you can come away from a session with F# with the impression that it's just being needlessly picky.

McCaffrey would have you believe that if you write this in C#, you are being explicit about your data types:

    int x = someFunction(y, z);
    
The type of x is int, right? Well, yes, but that doesn't mean that "someFunction" actually returned int. All this code tells you is that the compiler is somehow able to convert or coerce the function's return value to int. The C# type conversion rules are fairly complex, and can result in surprises. (http://goo.gl/OCU9xt)

The most common alternative in F# looks like this:

    let x = someFunction y z

Although it may look like a dynamically-typed version of the C# line above, it doesn't work like that at all. This code tells the compiler to figure out the actual return type of someFunction, then give x the exact same data type. Looking at the code, you may not immediately know what that data type is, but you do know that it is correct. You also know that that function isn't going to have a different return type in another context, something that can happen in a dynamically-typed programming language. In F#, once a function is known to return a particular data type, it always returns that particular data type, or it doesn't compile.

F# will let you be explicit about your data types, if you like:

    let x : int = someFunction y z

Although that may look more like the C# line above, it doesn't actually mean the same thing. What you've done now is asserted that someFunction returns int — not uint32, not int16, not byte. If it returns anything other than int, it won't compile. 

Sometimes explicit type annotation is necessary. Type inference isn't always perfect, so you do sometimes have to help the compiler choose the data type of a given expression.

This F# design choice gets you the expressive compactness of a dynamically typed programming language with all the safety that you get from a statically typed programming language. In fact, code in F# is usually safer than in most other statically typed programming languages. There are no silent data type conversions in F#, a feature that would have prevented the famous crash of Ariadne 5 Flight 501. (http://goo.gl/71tNM9) Another feature of F#, numeric unit enforcement, would have prevented a more recent space disaster, the crash of the Mars Climate Orbiter, traced to incorrect mixing of SI and imperial units. (http://goo.gl/WDqQBj) F# also won't let you implicitly mix integer and floating point values, another common programming language "feature" that's caused untold amounts of trouble. (http://floating-point-gui.de/)

In the end, I'm happy I learned F# even though I don't get to use it all that often. Learning any FP language will make you a better programmer, and I think F# makes a particularly good choice for getting that experience, if you haven't already. It strikes an uncommonly nice balance between concision, utility, and clarity. Even if you go back to imperative or OO programming, or end up using some other FP language, the experience you get from F# will continue to shape the way you approach programming.

Post has attachment
If you're a C4D user and have always applied materials to 3D models by projecting them straight onto the model using the standard methods — cubic, spherical, flat, etc. — you should check out this tutorial. It teaches you how to build your own UV maps from a 3D model, which gives you much greater control over how the textures get applied to the model.

I ran into the need when exporting a model from C4D to Photoshop, then trying to use Photoshop's 3D texture painting features. Photoshop creates its own UV maps from your model automatically if they don't have one associated already, and it did a terrible job in the case of this model. It made texture painting basically impossible. After manually unwrapping the UV in C4D, I was able to get Photoshop to paint sanely on the exact same model.

Good job, Joey!

Post has attachment
Ever heard the claim "Everything is a file in Unix"? It's one of those statements that isn't really true, even as it tries to frame a kind of Truth.

I've dissected this matter in some detail in an answer to a question on the Unix Stack Exchange site. I think it's an interesting read, if you use Linux, OS X, BSD, or big iron Unix.

Post has attachment
You come from a galaxy far, far away, on the t axis.
Photo

Post has attachment
I just had a thought about the new Adobe Creative Cloud licensing model that I hadn't seen mentioned anywhere.

With this move, Adobe has decoupled their sales from market feedback mechanisms. They have thus made themselves instantly less competitive.

Think about it: before this move, their Flash business was probably running a net loss to Adobe, which is why they now try to trick you into installing antivirus trialware and such. I'm guessing that the money in from Flash Professional isn't paying for all the bandwidth and development time required to keep Flash Player (and AIR) up to date. They tried to rescue the business with Flex (a.k.a. Flash Builder), but they abandoned that, too.

Now Adobe is moving into HTML5 development tools. As part of this switch, they've taken on new no-cost-software development burdens like PhoneGap, CreateJS, and Brackets.

(See http://html.adobe.com/opensource/)

If I'm right that Flash eventually turned into a net loss for Adobe, what happens when (!) that happens to HTML5, too? How will they know when to pull the plug on their free software efforts? They can no longer compare the cost of those efforts to the revenue from the product those efforts directly benefit. Their creative software revenue is now intermixed.

This is a small version of the problems that you get when you try to centrally plan an economy.

The point is, even if their Creative Cloud membership numbers rise to the point where the income from that matched the pre-CC revenue, they've left themselves more vulnerable to changing market conditions.

Adobe just tossed their Kryptonite out into a field, and are hoping no one will ever find it.

Post has attachment
Smug lizard, captured in Capitol Reef National Park, Utah
Photo

Post has attachment
I was writing a program with a recursive loop in it today, and ran into an amusing situation.

For the non-programmers reading this, a recursive loop is when a bit of code in a program calls itself repeatedly until some condition is satisfied. When writing such code, you ideally want to structure it so that each iteration requires no additional memory, since the size of the stack — where that memory comes from in most recursive programs — is typically limited to a few megabytes. The amount of memory your loop requires per iteration therefore effectively caps the number of times you can go through the loop. This isn't always possible.

What made the situation really interesting is that the number of iterations in this particular program was probabilistic. That is, you couldn't predict a priori how many iterations it would take. The outer condition had a 1 in 4 chance of being true, which would let it get to the inner condition, which had a 5 in 6 chance of sending program execution recursively back through the loop again. The total chance of going 'round the loop again was therefore a bit over 1 in 5. A 21% chance definitely sounds like something to worry about, until you then try to figure out how many times in a row that pair of events has to happen before disaster actually strikes. Mathematically, it is (1/4 ⨉ 5/6) ^ N, where N is the number of iterations it takes to smash the stack.

Say you pick N = 100 out of the air. That is, we'll say we can recursively go 100 levels deep before our program crashes due to running out of stack memory. That number is way, way too low in my particular case, but let's just go with it and see where that gets us.

Result? The chance of my program smashing the stack due to getting 100 levels deep before finishing is roughly 1 in 10 to the 70th power.

We computer geeks write that as 1e-70, which is shorthand for scientific notation, 1 ⨉ 10^-70, which is shorthand for 1 divided by 10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000.

So you see why we prefer to write "1e-70."

Just how tiny is 1e-70?

About a thousand of the 300 million United States citizens get killed by lightning every year. With 365 days per year, that means that your particular chance, dear reader, of being killed by lighting today is about 1e-8. That means the chance of my program running out of stack RAM today is roughly equal to your chance of being hit by a fatal lightning strike nine times. Today.

To put it another way, you are a thousand million billion trillion (1e30) times more likely to be killed by a lightning strike while you are playing poker in the back of your limousine having been dealt a straight flush on your way to the White House because you have been elected the next President of the United States. I said your limousine, because in this scenario you won $1,000,000 in the New Mexico State Lottery. Oh, and in the rear-view mirror of the limo are the smoking ruins of your house, which happened to have been hit by a meteor today.

This particular stack-up of low-probability events has a chance of occurring roughly equal to 1e30.

(It's actually 9e-9 ⨉ 5e-6 ⨉ 6e-5 ⨉ 1e-7 ⨉ 2e-15 = 5.4e-40, within an order of magnitude of 1e-70 ⨉ 1e30. Close enough.)

But is that the real probability of crashing the program?

If you assume a more reasonable value for N, say 10,000, then try to calculate it, your calculator will probably dissolve into an innumerate puddle, giving an answer of 0, that being as close as it can come to saying, "It ain't gonna happen." If you then drag out a big gun like Wolfram Alpha to get the real answer, you find the chances are on the order of 1e-6,814. That's math-speak for, "It ain't gonna happen. Truly."

If you liked this, I recommend you add XKCD's "What If" to your feed reader.

Post has attachment
There's a lot of info on the Internet about lossy video compression. (H.264, WebM, MPEG-2...)

Much sparser is information on lossless and effectively lossless video compression. Lossy video compression is great for delivery, but during testing and production, it has too many bad side effects.

I've written an article on creating low-loss video files using ffmpeg, which also goes into a lot of detail on what your options are. A 6 Mbit/s H.264 720p HD video file balloons to about 100x that size when you convert it to a truly uncompressed form, but there are lots of options between these data rates. One of them is almost certain to get you the results you're after without that 100x data rate hit.
Wait while more posts are being loaded