Experiences with the Foresight Exchange

Ken Kittlitz

The Idea of Idea Futures

Conclusive evidence of intelligent extraterrestrial life will be detected by 2050.

The major rationale for such a market is to provide what Hanson, writing on the subject of scientific research, calls an "honest consensus". In science, as in other areas, the attention an idea receives sometimes has less to do with its merit than with how well it meshes with conventional wisdom or pleases those in positions of power. Research funding is difficult to allocate objectively, but only partly because of the uncertainty inherent in evaluating any prospectus. Scientists are people, and their judgment is subject to self-interest and other biases, just like the rest of us.

Putting ideas into a marketplace recognizes this and gives people incentive to overcome it. Someone spending real money on an Idea Futures claim coupon has good reason to consider the claim's likelihood objectively, regardless of how they feel about it personally. The marketplace does not often reward those who trade based on wish-fulfillment or the desire to make a statement.

The Structure of an Idea Futures Market

• claims about future events;
• banks that issue coupon pairs (see "Trading", below) in claims in exchange for an asset (e.g., a dollar) and that redeem pairs for an asset;
• clearinghouses that match buy and sell orders for coupons;
• claim creators who sponsor new claims;
• traders who buy and sell coupons with each other;
• judges who judge a claim true or false at its due date.

Typically, costs will be associated with creating a claim; banks and clearinghouses may demand fees for supporting/listing it, and a judge having a reputation for fairness may expect remuneration for his/her efforts. Alternately, the judging fee may partly financed by a percentage of the claim's capitalization, or if necessary, by audit lotteries (Hanson, 1990).
 

Trading in an Idea Futures Market

1. The trader who placed the sell order buys the required number of coupon pairs from a bank for the price of one asset each. The coupons in each pair are termed YES and NO, representing a position "for" and "against" the claim, respectively.
2. The seller sells the YES coupons to the buyer at the agreed upon price.
3. The buyer now has the YES coupons, and the seller has the NO coupons.

Effectively, a trader who sells coupons she doesn't have is "shorting" the claim. However, unlike conventional market shorting, her gains or losses are bound by the range of possible coupon prices (0 to 100 percent of its underlying asset). Thus, the short seller can be required to maximally cover the short at the time of the trade.

Like most short sellers, Mary is hoping to sell high and later buy low.  Suppose the claim's price later drops, and she is able to buy ten YES coupons at $0.20 each (cost: $2.00). She now has ten coupon pairs which she can redeem at the bank for $1.00 each (income: $10.00). She thus made $8.00 net from this trade, and spent $6.50 in the trade with John, leaving her with a profit of $1.50.
 

Claim Judgment

Although claims with binary outcomes are the most common, scaled claims, in which coupon holders can receive a percentage of the coupon's asset (other than 0 or 100) are also possible. For example, a scaled claim may pay out YES coupons at $0.57, meaning that NO coupon holders receive $0.43 per coupon.
 

IF: The First WWW Implementation

Unfortunately, IF's prototype code did not scale well to the increasing number of users, leading to frustrating waits for order processing. As well, the group wished to commercialize the game, but this promised to be complicated because Sean and Mark had written it while employees of the Alberta Research Council, using that company's equipment. To avoid such potential conflicts of interest, and to address the deficiencies of the IF system, we decided to re-implement it. Design and implementation of the Foresight Exchange began in the fall of 1995.
 

Features of the Foresight Exchange

General Structure

In FX, a clearinghouse has only one database server, and it only accepts one request at a time, thus removing the need for locks and simplifying the caching of frequently-retrieved data. It also returns "raw" data, leaving post-processing and formatting  to client programs. Such programs do not connect to the database server directly, but instead go through a "connection server", a thin intermediate process that protects the database server from rogue clients. A connection server is created for each client, and stays alive as long as the client needs, but only one connection server can be talking to the database server at a time.

The FXTP Protocol

user_info 107,email,cash,score

The Economy of FX

Creating a claim costs the creator 50 credibills, both to discourage frivolous claims and acknowledge that claims would not be free in a real-money market. The judge of a claim gets 25 credibills when he/she judges the claim.

The Score

networth + feesOut
------------------
allowance + feesIn

An average score is 1.000; anything above that is good, anything below, bad.

Claim Categories & User Profiles

The Claim Creation Process

As in IF, claims about sporting events are disallowed, under the assumption that there are other ways to discover their odds. Most other subjects are fair game for claim creation, though the administrators reserve the right to disallow claims they think are in poor taste (e.g., "dead pool" claims).

Public Vs. Private Accounts

Mailing Lists

(To subscribe to one of these lists, send an email with the line "subscribe fx-discuss" [for example] in the body to majordomo@ideosphere.com.)

Experiences

Overall Growth of the System

Our expectation was that the greater functionality and scalability of FX would make up for the loss of user data, but many IF players saw instead that the nearly two years of effort they'd expended to create good scores and portfolios disappeared almost literally overnight. Bugs and other teething problems in the early weeks of FX's life aggravated the situation, and a significant number of IF players dropped out of the game. In retrospect, we should have warned the user community about the ramifications of the transition beforehand instead of acting unilaterally, a mistake we've tried to avoid repeating.

In general, FX's growth has been considerably slower than that of the original game. Trading frequency dropped sharply after the transition, and only fairly recently has started to rival that of IF. Over 2500 players are now registered, but many never trade or trade only a few times before losing interest.  Though this problem also afflicted IF, FX's population growth is much flatter.

There are several reasons for this. When IF debuted in 1994, it quite likely was the first trading system on the Internet -- truly an accomplishment. That the commodities traded were different from those of conventional markets only added to its appeal. Since then, the number of high-quality sites has exploded; several have Idea Futures-like games -- offering prizes to successful traders -- and there are numerous discount brokerages and online gambling sites that allow people to invest and/or wager real money. And all of these are but a fraction of the diversions available.

Compared to many sites, FX looks plain. Its user interface is designed to be functional rather than pretty, and can appear confusing and intimidating to the new user. The mechanics of the market, particularly the concept of "NO" coupons, are not very intuitive and can be difficult to understand (people either seem to grasp the idea immediately or have a great deal of trouble with it, and a background in finance or mathematics doesn't necessarily mean a person will fall into the first category). Achieving a good score and the respect of the community are the only rewards for trading well.

Viewed from this perspective, it's hardly surprising that the FX community is small. On the other hand, the kind of people that are interested in the game are non-average indeed, possessing a diverse range of knowledge and experience. Fx-discuss rarely suffers from off-topic or argumentative posts, and makes for fascinating reading.  If some of the other Idea Futures sites (listed on Hanson's Idea Futures page), with claims about sporting events and celebrity popularity, can be termed low-brow, then FX definitely takes the high-brow approach.

A final, more prosaic, reason why trading and claim creation is more sporadic on FX is because of the limited economy. Unlike IF, FX originally had an allowance cap of 1000 credibills, paid out weekly in installments of 50. Players tend to become less likely to create new claims and trade speculatively when they are no longer receiving a regular influx of cash. The cap was increased to 2000 credibills about a year ago to stimulate trade and claim creation, and people have suggested that the cap be removed entirely.

It may be removed at some point, but others have argued that the limited allowance has made them better traders than they would be if they could always rely on receiving fresh capital to work with. As well, not having a cap tends to dilute the effects of a player's actions. With one in place, "mistakes are more strongly disciplined and good guesses are more strongly rewarded" (Karl Evan Hallowell, personal communication).
 

FX's Predictive Success and Responsiveness

My findings regarding FX involved 36 of the originally retired claims.  I  limited the sample set specifically to claims that had an adequate number of people and transactions (statistically speaking).  I asked the question "How often was the market right compared to how often the market thought it would be right?".  That is, did the prices actually realistically represent the probability of the claim being paid.  I "folded" the prices so that I grouped 90's with 10's and 80's with 20's, etc. I then took the average folded price - about 75% - and compared it to how often the market was "right".  I considered a claim to pay as "the market expected" when the yes coupon paid off if the price of the yes coupon was over 50 or when the no coupon paid off if the price of the yes coupon was less than 50.  I found that the average "folded" price (where 65 is considered equal to 35) to be about 74%.  In other words, the market was right just about as often as it expected to be right.  This is very well calibrated by any measure.  However, this averaging hides a more interesting phenomenon.

I found an interesting phenomenon that indicated that the market was not perfectly calibrated.  When I looked at all the times the price of any option was, say, 90 I would find that it typically had a higher than 90% chance of paying the yes coupon.  And if an option was selling at 10 it typically had less than a 10% chance of paying the yes coupon.  This was true for all prices that were close to either 100 or 0.  This seems to indicate that the high payoff bets are overpriced and that the safe but low payoff are underpriced.  It seems that this group of players are willing to pay a premium for a chance for a major score like winning a $1 payoff that you only paid $.03 for.  The sample size was not sufficient to accurately estimate the magnitude of this effect but it was sufficiently large to show there is such an effect." (personal communication)
 

Hal Finney argued that the "50 cent bias", if it really existed, might not be a true bias at all, but instead a reflection of market uncertainty. To further characterize the phenomenon, Hal proposed the CRAK claim, which is to be judged true if any of six "crackpot" (i.e., very unlikely) claims comes true. His reasoning was:

Suppose that there is a 50-cent bias, and that each of the crackpot claims should have a true probability of only .01, say, but IF is
consistently pricing it at 5-10 due to the bias.  Now the probability of Crak being true would be 1-(.99**6) or about 6%, so the price of
Crak should be about 6.  But if there is no such bias, and the prices of these various claims represent consensus market probabilities, then the probability of Crak would be more like 1-(.93**6) or .35. [post to if-discuss mailing list, May 28, 1996]
 

(One more note on CRAK: the price soared in mid 1997, because a player announced he'd found a loophole in the wording of one of its constituent claims, Uran, that would allow an immediate YES judgement of both claims. The judge disagreed, and the price dropped back accordingly.)

As was the case with IF, the market incorporates new information very quickly. For example, the claim XLif (Extraterrestrial Life by 2050), reacted extremely quickly to NASA's announcement in August, 1996, that a rock originating from Mars contained evidence of organic compounds (see price plot).
.

Even with a small number of traders, trading play money, FX behaves very much like a real market.

Claim Creation and Judgement: Still an Art

Nonetheless, creating good claims is still very difficult, the most difficult part of the game, according to many players. The basic problem is that a claim writer is faced with two somewhat contradictory requirements: conciseness and precision. A claim with a long, detailed description is less likely to attract players than one that seems simple; unfortunately, simple claims with "obvious" criteria for judgement often turn out to be not so straightforward. Time and again, situations or ambiguities that the claim creator didn't foresee have been noted by the community, placing the onus on the judge to clarify his/her interpretation of the claim.

To facilitate this, the concept of a judge's statement was added to the game about eight months after it went live. A claim's judge can create and update a statement about how he/she intends to judge the claim; the statement appears on the claim's detail page. This has helped clarify many claims, but even judge's statements can be ambiguous. A fairly common statement, referred to by some as the "Standard Ambiguity Clause", has emerged:
 

Standard ambiguity clause: If the wording is found to be ambiguous I will judge on the basis of the obvious intent. If the intent is
ambiguous I will judge on the basis of precise wording. If both are ambiguous or if both are clear but conflict, I will look for a
solution that causes the least damage to FX as a market and game.

Those who prefer judgements based on intent argue that the meaning, or "spirit", of a claim is usually obvious, and that its judgement should reflect this meaning, not some technicality or loophole caused by ambiguous wording. Those who support judgement based on wording alone reply that there is often more than one "obvious" intent of the claim's wording; a judgement that one player would consider to be in the spirit of the claim might surprise and anger another. They believe that the claim's literal wording is the only common ground for judgement, even if the results are not would the creator originally intended.

Of course, proponents of the claim's "spirit" would counter-argue that there is no such thing as the "literal" meaning of a claim: by definition, the reader interprets the words of its description, and interpretations may differ. The debate has been vigorous, with no consensus emerging. At this point, it's not clear which approach, if either, is desirable.

To be fair, many of the claims that have sparked debate over how they should be judged were those imported from IF; FX claims are generally much better-worded, thanks largely to the new approval process. But though this has lessened the problem, it hasn't eliminated it.

Claim Trends

To some extent, claims created in FX continue this trend. Many are scientific in nature or involve the proof/disproof of mathematical conjectures. Financial claims such as  DJIA (which seeks to predict the year-end Down Jones Industrial Average) generally attract interest, perhaps because they are usually unambiguous.  Some players argue that because other mechanisms -- the real stock markets -- effectively predict the same thing, such claims should be lumped in with sports claims and disallowed. Be that as it may, these claims have not yet proliferated to the extent of cluttering the exchange, and so don't seem pose a serious problem.

Disaster-based claims seem to be on the decline, though that may be because most of the obvious ones were created in the IF days. A number of Y2K claims have emerged in the past year or so:
 

Y2Kdth: >=200 deaths caused by Y2K bug
Y2Kstk: Y2K closes US Stock trading
Y2Kcsh: Y2K-Fed Rolls out Presses
 

(A complete list of claims can be found here.)

Miscellaneous Topics

Cheating and Collusion

This is almost entirely an artifact of FX's play money economy -- the costs to the "sacrificial" players are negligible compared to what they would be in real money market, after all, and a real money market is likely to be less thin, meaning that other investors could intervene and profit from the collusion attempt (e.g., by trading with the sacrificial accounts).. FX's score algorithm, at best a crude approximation of trading talent, is taken seriously enough that some players wish to boot their score artificially. There have been a number of collusion incidences, some quite subtle and involving many accounts. Because a unique email address is all that's required to open an account, it's difficult to limit players to just one, despite warnings on the account creation page.

Fortunately, FX's policy of an "open" ticker (the IDs of the buyer and seller appear on each ticker line) allows the user community to watch for such activity. All instances of cheating/collusion detected so far have been discovered by observant players, not the game's administrators. Owners of suspect accounts are sent an email describing the situation and are asked to explain themselves. If no explanation is forthcoming (and none usually is) the accounts are suspended.

Cheating thus represents one of the imperfections of FX's attempt to accurately simulate an Idea Futures market, but at the same time shows that the market can be self-policing.

Automated Agents

More recently, Neal Gafter has introduced a "market maker" 'bot based on Hanson's design. The market maker attempts to improve liquidity by placing both buy and sell orders in a claim to narrow the price spread, but is designed such that players cannot exploit it (e.g., force it to consistently lose money).  At this point, the 'bot hasn't been in use long enough to tell if it's really succeeding at its goal.

Trading Strategies

Trading in short-term claims is often more intense, because players know that the claim will be resolved relatively quickly. For volatile claims, one strategy is to place both buy and sell orders in the hopes of benefiting from price fluctuations in the claim, a practice that generally requires more time spent monitoring price changes than does a simple "buy and hold" strategy. It's also perfectly legal to place orders, then reveal new information about the claim via a mailing list post. After all, what's the advantage of inside information if you can't put it to good use?

Another strategy, first popularized in IF, is to pick a thinly traded claim and boost the price up or down (depending on whether you're buying or selling). Since claim holdings are valued on last trade price, this activity can increase your score, at least temporarily. Usually, however, other players will act to push the claim price back to what they think is reasonable.

Research and Associated Projects

Douglas Hubbard of Hubbard Ross, has invented a system called Applied Information Economics for helping executives make IT decisions, and has worked with such firms as Entergy, the Giga Information Group and the European Space Agency. He is interested in Idea Futures markets as a means for capturing probabilities of various outcomes. Douglas did an analysis of retired claims to gauge the accuracy of the market.

Vladimir Nuri's Memelink is a site for tracking the popularity of memes as they move through the 'net. He considers FX to be an interesting meme, and we arranged a cross-link between the sites. FX claims that have been proposed but not yet activated (e.g., those needing judges) are listed on a Memelink page, allowing people to vote for claims they would like to see open for trading. The votes can only serve as rough indication of preferences, since there is no way of preventing people from voting multiple times, but it's obvious from looking at the tallies that some claims are more popular than others.

Future Plans

• voting system: Players often have suggestions for rule changes or new features, and these are frequently discussed on the mailing lists. Alas, consensus is rarely achieved. To help prioritize development efforts we introduced an Approval voting page at the beginning of the year. The first, "meta", vote was to help decide what topics future votes should be about. Players are allowed to create options, and vote for all options they support (a key feature of Approval voting). The winner was "rewards for claim owners/judges", and was the subject of the second vote. At this point, it's unclear whether the current system  is an improvement over the mailing lists -- over forty options on the topic were proposed! The vote results are understood to be only advisory, and it's likely that further work will go into the voting implementation before long.
• re-implementation: Although the software shows no signs of reaching the limits of its performance, a redesign is in the works to address some of its deficiencies, which include lack of a good query facility, reliance on the UNIX operating system (some would hardly consider that a deficiency) and loss of code organization after years of feature additions and re-development. When the new version is ready, existing user data will be ported to it, the goal being a smooth, painless transition, precisely what did not occur when IF changed to FX.
• trials in industry and academia: several companies and universities are using the software on a trial basis is a probability-gathering tool. Despite the unfamiliarity of the concepts, and the requirement of a relatively large number of users for accurate markets, some of the trials have yielded encouraging results.
• a real-money market: the true test of an Idea Futures market would come when real money is used to invest in the games, but the murky legality of such an enterprise is certainly a worry. However, we remain alert for ways of making it come about...

Acknowledgements