Mark Forster's Blog - Comments

Mark Forster comments on Oops!

Mark Forster — Thu, 16 Feb 2012 14:38:55 +0000

Mauricio:

It looks interesting but how long does the course go on for? I don't want the FV put off even longer!

Mark Forster comments on The Final Version - first look

Mark Forster — Thu, 16 Feb 2012 13:17:01 +0000

I'm no mathematician but surely it's easy to establish the exact odds between Colley's rule and the Rule of Three without having to run 100,000 trials or indeed any at all.

There are only six possible situations in the Colley v. R3 stand-off:

Each of the first three numbers must be either H=Highest; M=Middle; or L=Lowest

The possible combinations are:

L M H = R3 wins
M L H = Draw
M H L = Draw
L H M = Draw
H M L = Colley wins
H L M = Colley wins

Therefore:

1) The chances of both methods producing the same result are 50%.

2) Colley is twice as likely as R3 to produce a better result.

However there is a complicating factor which is what happens if the first number is the maximum. In this case some of Colley's wins will become draws. The likelihood of this happening is greater the lower the total number of numbers. With very a low number it would impact severely on Colley's chances of producing a better result than R3 (though only because both would produce the best available result).

My experimental results with the random generator seem to be in conformity with this.

So my conclusions are:

1) If you know there are only a small number of candidates then it doesn't matter much what method you use since you can probably afford the time to look at all of them anyway.

2) If you know there are a more than a small number of candidates, use Colley's rule.

3) If you don't know how many candidates there are, use Colley's rule.

4) If you can't go back or it would be inconvenient, use Colley's rule.

5) Precision and accuracy aren't the same thing.

Mauricio comments on Oops!

Mauricio — Thu, 16 Feb 2012 12:31:54 +0000

Hey Mark,

About making FV even better, I just came across a free online course from a Professor at the University of Michigan about using Models to make optimized decisions. I thought you might be interested in going through it, as it might affect positively the Final Version!

The previews look majestic! You can check them out here http://www.coursera.org/modelthinking/lecture/preview

And join here http://www.modelthinker-class.org/ ;-)

Mark Forster comments on The Final Version - first look

Mark Forster — Thu, 16 Feb 2012 12:23:06 +0000

I've now run a further 10 trials with the generator set at 1-20. Duplicates removed as before. The results were that Colley produced a better result 4 times, the Rule of 3 once, and they produced the same result on the remaining 5 occasions.

There seems to be some discrepancy between my results and yours. So far on the various settings which I've used Colley has produced 8 better results to the Rule of 3's two.

Mark Forster comments on The Final Version - first look

Mark Forster — Thu, 16 Feb 2012 11:58:52 +0000

Here's the numbers for the above experiment:

1-100:

33 37 9 (Both 37)
100 + another 99 numbers (Both 100)
60 34 49 20 42 7 76 (R3 60; Colley 76)
66 53 14 87 (R3 66; Colley 87)
80 16 7 16 78 55 89 (R3 80; Colley 89)

(Colley 3, R3 0, Draw 2)

1-10:

3 4 10 (R3 10; Colley 4)
2 1 5 (Both 5)
6 9 3 (Both 9)
10 + another 9 numbers (Both 10)
7 3 6 5 4 9 (R3 7; Colley 9)

(Colley 1, R3 1, Draw 3)

Mark Forster comments on The Final Version - first look

Mark Forster — Thu, 16 Feb 2012 11:43:42 +0000

FSE:

I haven't done 100,000 trials, but I tried using the random number generator at http://www.random.org/ to see what happened. I ran 5 trials which the generator set at 1-100 and 5 trials at 1-10. Duplicate numbers were removed.

In only one out of the 10 trials did the Rule of Three produce a better result than Colley's. That was with the 1-10 range setting. In five cases both methods produced the same result. And four results were better using Colley's rule.

As far as time goes, the rule of three was obviously faster most of the time (it would be difficult to see how it couldn't be). Even so Colley's rule produced the same result faster on two occasions.

Maybe I was just lucky.

FSE comments on The Final Version - first look

FSE — Thu, 16 Feb 2012 06:16:47 +0000

Bernie and Mark,

Ok, I just ran some simulations in MATLAB for this problem. Yes, I used 100,000 trials per simulation!

Rather than implementing a coin-tossing scheme, I used a very minimalist alternative to Colley's Rule. Let's call it "The Rule of Three":

"Look only at the first three possibilities. Choose the best one among those three."

It's painfully simple, but believe it or not "The Rule of Three" proved superior to Colley's Rule regardless of the total number of possibilities. I can post a detailed table of results if you want, but here is the executive summary:

1) When there is a small number of possibilities (i.e. between 5 and 20), "The Rule of Three" provides much better results on average than Colley's Rule and takes slightly less time on average.

2) When there is a large number of possibilities (i.e. between 20 and 10,000), "The Rule of Three" provides slightly better results than Colley's rule, and takes much less time.

I have to say that the results somewhat surprised me, but there they are. This tends to reaffirm my suspicion that Colley's Rule was meant to be used in a different setting.

Bernie comments on The Final Version - first look

Bernie — Thu, 16 Feb 2012 02:10:20 +0000

FSE wrote:
<>

Not at all, FSE. I describe the analysis in detail on page 1 of these replies. I cannot find a way to link directly to my entry from here, but you can find it by clicking back to page 1 and scrolling about 2/3 of the way down. There are a few followups shortly after that shed some more light with a more concrete explanation. Please have a look and let me know if you find errors. We'd better take that to the forum, though, rather than cluttering up this blog post with more equations. I will gladly join you on a new forum thread to iron it out.

Maybe you are looking at the expression 1-x*x and being reminded of the math that arises from a straightforward lottery? E.g., 1-0.8*0.8 does happen to be our probability of winning a 1-of-5 lottery if we are given two chances at it, but that is nothing like the reasoning used in my analysis. The resemblance is pure coincidence.

Instead, I have essentially asked the following:

If 100,000 people, completely independently, made selections by Colley's Rule, what would the luckiest 37,000 people's results look like? The answer is that they would all obtain results in their top quintile of desirable options. I chose 37% in order to compare to the Secretary's case, in which we've learned that the luckiest 37,000 people would actually obtain their very best possible option.

<<... Colley is no better than flipping a coin every so often and ending your search when it lands heads.>>

Colley's is *vastly* better than this! Hopefully a look at the analysis and other comments on page 1 will clear that up. Since Colley's is very effective for small N, you can also confirm this through experiment: write numbers on index cards, shuffle them into a stack, benchmark the top number, and proceed down the stack until you beat it. Make a histogram of your results, and they should beat the pants off of the occasional coin flip or a straight lottery, or even a best of two lotteries ... but I don't recommend 100,000 trials!

Mark Forster comments on The Final Version - first look

Mark Forster — Thu, 16 Feb 2012 01:16:49 +0000

FSE:

The purpose of Colley's rule is to save time. It gives you a simple method of getting a good result in as little time as possible. And you don't need to know how many candidates are available for it to work.

I don't see how anything you've said affects that.

Mark Forster comments on The Final Version - first look

Mark Forster — Thu, 16 Feb 2012 01:07:55 +0000

Before you all get into even greater flights of fancy, I'll just remind people of my earlier comment that the FV has changed considerably while I've been working on it so you shouldn't assume that earlier descriptions of it are still valid.