Our benchmark: where we are and where we're going

Datacircle is a new company. Our benchmark is still small.

Now: one public benchmark

Our inputs are public. Each provider can see where their API failed, call by call. We're adding as many providers as we can to the public benchmark.

Providers can game a benchmark

A provider can tune their API to a public list of inputs. Hiding the inputs doesn't stop this. We call each provider with our own API keys, so each provider sees our queries on their own account. From those queries, a provider can work out where they fail. An anonymous benchmark has the same problem.

Next: an anonymous benchmark

An anonymous benchmark keeps the inputs private. Once enough providers in a category do well on the public benchmark, we'll write an anonymous benchmark for that category.

The end goal: a continuous benchmark

A continuous benchmark works best. We put a proxy in front of each provider. We score the real requests that pass through and the answers that come back. The proxy is hard to build. We'll build one after the anonymous benchmark.