We’re happy to announce that Round 17 of the TechEmpower Framework Benchmarks project is now available. Since the adoption of Continuous Benchmarking, the creation of an official Round is a fairly simple process:

  1. Try to reduce errors in framework implementations. We want an official Round to have a solid showing by as many frameworks as feasible given limited personnel bandwidth.
  2. Select a continuous run on the physical hardware (named “Citrine”) that looks good and identify its Git commit.
  3. Run the same commit on cloud (Azure).
  4. Write a blog entry and post the Round.

For weeks, we have been stuck at step 4 waiting on me to write something, so I’m going to keep it short and sweet to get this out before Halloween.

Stratified database results

As you review Round 17 results, you’ll notice that Postgres database tests are stratified—there are groups of test implementations that seem implausibly faster than other test implementations.

The underlying cause of this is use of a Postgres protocol feature we have characterized as “query pipelining” because it is conceptually similar to HTTP pipelining. We call it pipelining, but you could also call it multiplexing. It’s a feature of the “Extended Query” protocol in Postgres. Query pipelining allows database clients to send multiple queries without needing to wait for each response before sending the next. It’s similar to batching but provided invisibly in the driver. Thereotically (and we believe in practice) short queries could be sent together in the same network packets.

Importantly (at least for us), the client application’s business logic is unchanged. This is not batching of queries implemented by the application or web frameworks, but rather an optimization provided invisibly by the database driver.

We discussed the pros and cons of permitting this optimization in our tests. While we have disallowed several performance-optimizing tactics elsewhere as violations of the rules or spirit of our tests, we ultimately arrived at permitting this optimization for two reasons. First, it can be applied “seamlessly” to application code. The application code does not need to be aware this is happening. And second, because this isn’t a trick applied at the application tier, but rather an optimization that potentially benefits all applications (once implemented by drivers), we feel this is precisely the sort of improvement this project should encourage.

In short, we think this feature in the Postgres wire protocol is great. We suspect that over time, more platforms will support the “pipelining” capability, gradually re-balancing the stratification we’re seeing.

The effect of this feature is most emphatic on the Multi-query and Updates tests, both of which execute a tremendous number of queries per second. We are also considering adding an attribute to indicate and filter test implementations using traditional database connections versus those using pipelining.

Other updates

  • Round 17 is now measuring 179 frameworks.
  • New languages such as F# are now represented.
  • The results web site is a bit wider in response to feedback.

In the works

We are presently working on a few things that we hope to share with the community soon. These include:

  • Potential changes to the physical environment’s network to allow further differentiation of ultra high-performance frameworks on the Plaintext test in particular.
  • Badges for framework maintainers to share and celebrate their performance tier, similar to popular continuous integration badges seen in GitHub repositories.

Notes

Thank you to all contributors and fans of this project. As always, we really appreciate your continued interest, feedback, and patience!

Round 17 is composed of: