Why is it that the standard of proof in software development theory is “Some famous person said it’s cool and I can think of an analogy to a Malcolm Gladwell book so therefore i’m right”?

• Standards of proof in medicine—scurvy, limeys
• Control groups and test groups.
• Yes, all studies are flawed.
• My Comment: The difference between nitpicking and valid criticism of a study is this. Any study is going to miss some data points (patchy, piecemeal coverage of the space we’re interested in) and the measurements are going to be wrong ε. ε may even be piped through a function (automorphism) such that the whole study is ruined. (for example you measured the wrong people or you measured the wrong way or you introduced bias or you measured a special proper subset but wrongly generalised the special property to the general case)

  We know this going in. The challenge is to distinguish ε’s that are deadly to the conclusion from epsilon;’s that are inconvenient and imperfect but don’t phase-change the conclusion. (think bounded perturbation versus sign change)

  This is why it’s nice to be able to make arguments using order-of-magnitude or sign. If you observe multiple orders of magnitude difference in beta; between the control group and the test group, then as long as your sample wasn’t horrifically terrible, the conclusion that A>B is still going to hold up.

• Rank speculation versus data.
• http://cochrane.org. Look at the data yourself. Do you see a connexion between vaccination and autism?
• “It’s almost impossible to get a paper published in a top journal without having actually tried it in the real world.”
• “All the work that’s been done to date on [estimating software development time] is pretty much worthless. … The engineers are just going to tell us what they think we want to hear.” (or their answer will be driven by anchoring effects)
• Garbage in, garbage out. (He gives an example where some garbage estimates of how long a software project would take are used as the base data, before being thrown through a gleaming shimmering algorithm that makes them look meaningful.)
• “If [developers are more accurate estimating how long a project will take on an hours-scale than a months-scale,] then that’s a powerful argument for using agile methods. I’ve heard many people make that argument, but we don’t have any data to back it up.”
• You’ve all heard “Great programmers are 28 times more productive than OK programmers. Or 40, 50, 100. I just pick a number that’s big enough to be impressive but not so large that you’re going to doubt me.”
• My question: I would love to see a #BigData person scrape the web for all such claims.
• “All of the claims you’re reading about the relative productivity of programmers can be traced back to: 12 people in 1968 [batch programming] for an afternoon, or 54 people [in the 1980’s] for an hour. How confident are you in those claims now?”
• “The best programmer I ever met — his only higher education was two years at a rabbinical college.”
• Claims he could take $150M in research money and make 5% of $1,000B. (A classic appeal to the “No number can be less than 1%” theorem. Textbook VC-pitch logic.)
• Klaus _____; “Regardless of language” (scheme, assembly, java, python, …) “programmers produce the same number of lines-of-code per hour.”
• Boehm 1975: “Most bugs are introduced in the design & analysis phase.”
• “The sooner a bug is caught in the development process, the less it costs to fix” (he seems to indicate the cost growth is exponential with time)
• “Adding a feature doubles production time. (Conversely, if the developers can say “no” to a few features, development time goes down convexly.)”
• “If you have to rewrite more than 25% of the code, you’re better off starting from scratch.”
• Minute 33: “Hour for hour, the fastest way to fix code is to read it. Not to run it. Not to write unit tests. Sitting down and having someone else go through your code. 60%-90% of bugs can be removed before the code is even run.”
• “But it turns out that most of the value comes from the first reader in the first hour of looking at the code.” That’s a couple hundred lines.
• Conway’s Law is true.
• A big-data statistical analysis of how Windows Vista was built, trying to find regressors that predict the fault rate.

1. Physical co-location of programmers did not matter.
2. Distance in organisational chart does matter.

• Stereotypical anti-religious view of scientific progress. (“The difference between science and religion is…”). Blah. Not very evidence-based in an evidence-based lecture.
• “Ask a successful start-up what’s the reason for their success and they’re going to get it wrong.” Up to academics to figure out what makes for success across various cases. (I agree; this is the point of management science. And he mentions the Harvard Business Review as a touchstone.)
• Rob Pike: “I don’t think I’ve ever seen beautiful code.”
• Typical self-perception of a rich programmer / white collar / professor. Whatev.
• Some words about becoming an adult. Someday there will be no higher authority you can appeal to other than the people who were 18 when you were.
• Making decisions when nobody knows the necessary information. (Seems out of place in a talk where he acts like his views are all backed-up by data. Whatever, though. I think we can all agree that being right is better than being wrong and that one should change one’s opinion whenever “objective evidence says you’re wrong” (whatever that means). And, obviously, as a leader you usually don’t get as much information as you would like—but you have to decide something anyway.)
• The difference between the Bolsheviks and the Trotskyists is that whilst Bolsheiks believe the masses have to take to the streets to effect change, Trotskyists believe a handful of focussed people who get on the right committees can change the world. Examples: the teaching of evolution; abortion. Don’t write a blog (oops) or start a Facebook group to effect political change, go to the pressure point. Put on a suit, try to sound like an adult, and make your case calmly and rationally to the people in charge.

Hat tip to @gnycl.

(Source: http://player.vimeo.com/)