Tech lessons learned from the wisdom of crowds

(continued from previous page)

Each "trader" received $50 in their account to start with, and was told that the more accurate their prediction, the more money they would make. The market opened with an initial price of on-time delivery set to 16 2/3 cents.

"The price of 'before November' dropped to zero right away," Proebsting said. "The price of 'on time' in about two to three minutes dropped to 2.3 cents on the dollar." Translated, that's more than 30-to-1 odds against on-time delivery.

Then the woman who was responsible for scheduling started trying to convince her colleagues who were buying and selling future delivery dates. "She was able to talk (on-time delivery) up to around 3 cents," Proebsting said. "People really enjoyed moving the price...They loved this."

"The next day the director comes into my office and said, 'What have you done?'" Proebsting said. But further investigation showed that the product actually was behind schedule, even though nobody was telling management, and it eventually shipped in February.

Policy analysis markets: Chris Hibbert is an inventor and programmer who's writing open-source software to build a prediction market. It's called Zocalo.

Prediction software should work with play money, real money and an internal corporate currency that might be reputation-based, Hibbert said.

Most existing prediction markets like the Hollywood Stock Exchange and Tradesports.com focus too much on topics like sports and movie success, when policy and political questions can be more interesting, Hibbert said.

Robin Hanson, an associate professor of economics at George Mason University, is the closest that prediction markets have to a founding father. But he became a politically controversial one when his idea of a policy analysis market that could, for instance, help to predict terrorist attacks was savaged by Democratic senators eager to take a swipe at the Republican administration that funded it.

"The example that you provide in your report would let participants gamble on the question, 'Will terrorists attack Israel with bioweapons in the next year?'" Sens. Ron Wyden, D-Ore., and Byron Dorgan, D-N.D., wrote to the Defense Advanced Research Projects Agency in 2003. "Surely, such a threat should be met with intelligence gathering of the highest quality--not by putting the question to individuals betting on an Internet Web site."

On Wednesday, Hanson quipped that the flap wasn't the publicity "we would have asked for in starting an industry." A paper he wrote, though, showed that more informed journalists and more recent news articles were more likely to give a positive impression of policy analysis markets (PDF).

Hanson said that conditional probabilities were worthy of more experimentation: If a company switched ad agencies, would it increase revenue? If the U.S. provides cash subsidies to Jordan, will that nation have a better economy or a worse one?

"The market is telling you the consequences of your actions," Hanson said. "If you do this, what are the consequences?"

[kdewalt]

The Wisdom of Development Teams

As I discuss in http://kevindewalt.com/blog/2006/12/26/the-wisdom-of-development-teams/, it shocks me that companies such as Microsoft actually think that they need sophisticated schemes like this to figure out if software will ship late.

My prediction system for determining when software will ship:

Ask the development team.

You see, software doesn?t build itself. People build it. People who usually know what they are doing and are very close to the problems. Unfortunately, the leadership at big companies isn?t close to the problems. Leadership ?knows the big picture? - like dates, budget, and sometimes the name. Maybe even the scope. But more than anything else, leadership knows the date. ?The next release will happen on June 1st!?

Why do they know the date? The date for delivery in big companies is usually decided in management meetings. Sometimes management gets input from the toiling masses, sometimes not. At these sort of meetings the project manager usually gets persuaded into committing to a date. The circumstances around these types of decisions and the unfortunate impact on software development teams is eloquently discussed in Death March by Edward Yourdon.

In any case, here?s the reality: the developers, testers, and others on the team know whether a company is going to hit the date. Unfortunately, nobody asks them because management doesn?t want to hear the truth. It happens every day in every large company. Leaders don?t want bad news, they want to believe that everything will work out according to the agreement at the meeting.

Ok, so you?re in management at a large company. Hopefully you?ve had some experience as a developer or software project manager. You?ve felt the pain of tradeoffs between scope and time and you know that technology has a funny way of not working very well. In any case, before you suggest building some innovative market prediction system, try strolling down the hall and talking to the development team. They may be very reluctant to talk to you at first, but if you ask them questions about the challenges and problems on the project I guarantee they will open up. They probably won?t tell you explicitly if the project will be late, but you will be able to figure it out for yourself with some patience and careful listening.

I guarantee you that you?ll quickly realize that the problem in most corporate development efforts is not the absence of a sophisticated prediction system: it is the lack of trust and frank dialog between developers and management.

[kdewalt]

Broken link

Broken link above...

http://kevindewalt.com/blog/2006/12/26/the-wisdom-of-development-teams/