We’re pleased to announce Round 22 of the TechEmpower Framework Benchmarks!

The TechEmpower Framework Benchmarks project celebrates its 10th anniversary, boasting significant engagement with over 7,000 stars on GitHub and more than 7,100 Pull Requests. Renowned as one of the leading projects of its kind, it benchmarks the peak performance of server-side web application frameworks and platforms, primarily using tests contributed by the community. Numerous individuals and organizations leverage the insights from The TechEmpower Framework Benchmarks to enhance their framework’s performance.

Microsoft has been steadfast in their dedication to improving the performance of their .NET framework, and has been active in the Framework Benchmark community to further this goal. With the announcement of the release of .NET 8, it is clear that performance is paramount.

Here are some updates from our contributors

@franz1981 on GitHub, @forked_franz on Twitter:

Oliver Trosien says:

@fundon

@fakeshadow says in response:

@synopse

Notes

A heartfelt thank you to all our contributors and fans! We recognize the complexities involved in executing a benchmarking project accurately. While it’s challenging to meet everyone’s expectations, we are committed to continual improvement and innovation, made possible with your invaluable support and collaboration.

Round 22 is composed of:

Run ID: 66d86090-b6d0-46b3-9752-5aa4913b2e33 on our Citrine environment.

Follow us on Twitter for regular updates.

Today we announce the results of the twentieth official round of the TechEmpower Framework Benchmarks project.

Now in its eighth year, this project measures the high-water mark performance of server side web application frameworks and platforms using predominantly community-contributed test implementations. The project has processed more than 5,200 pull requests from contributors.

Round 20 Updates from our contributors

In the months between Round 19 and Round 20, about four hundred pull requests were processed. Some highlights shared by our contributors:

• @matt-42 says Lithium has optimized its Postgres implementation.
• @dantti created the ASql library to talk to Postgres in async mode and put this to use with Cutelyst.
• @sebastienros reports that Stephen Toub posted an excellent blog containing detail on all the performance improvements in .NET 5.
• @sagenschneider reports that OfficeFloor’s new use of asynchronous programming in its inversion of coupling control has enabled using reactive libraries such as R2DBC. More information can be found in OfficeFloor’s blog post about round 20.

(Please reach out if you are a contributor and didn’t yet get a chance to share your updates. We’ll get them added here.)

Notes

Thanks again to contributors and fans of this project! As always, we really appreciate your continued interest, feedback, and patience!

Round 20 is composed of:

• Run 68ea3e13-def2-484a-b832-40fbcb46be61 from Azure.
• Run f78282e3-ed37-4e7c-9862-3163de23b23b from Citrine.

This project measures the high-water mark performance of server side web application frameworks and platforms using predominantly community-contributed test implementations. Since its inception as an open source project in 2013, community contributions have been numerous and continuous. Today, at the launch of Round 19, the project has processed more than 4,600 pull requests!

We can also measure the breadth of the project using time. We continuously run the benchmark suite, and each full run now takes approximately 111 hours (4.6 days) to execute the current suite of 2,625 tests. And that number continues to steadily grow, as we receive further test implementations

Composite scores and TPR

Round 19 introduces two new features in the results web site: Composite scores and a hardware environment score we’re calling the TechEmpower Performance Rating (TPR). Both are available on the Composite scores tab for Rounds 19 and beyond.

Composite scores

Frameworks for which we have full test coverage will now have composite scores, which reflect an overall performance score across the project’s test types: JSON serialization, Single-query, Multi-query, Updates, Fortunes, and Plaintext. For each round, we normalize results for each test type and then apply subjective weights for each (e.g., we have given Fortunes a higher weight than Plaintext because Fortunes is a more realistic test type).

When additional test types are added, frameworks will need to include implementations of these test types to be included in the composite score chart.

You can read more about composite scores at the GitHub wiki.

TechEmpower Performance Rating (TPR)

With the composite scores described above, we are now able to use web application frameworks to measure the performance of hardware environments. This is an exploration of a new use-case for this project that is unrelated to the original goal of improving software performance. We believe this could be an interesting measure of hardware environment performance because it’s a holistic test of compute and network capacity, and based on a wide spectrum of software platforms and frameworks used in the creation of real-world applications. We look forward to your feedback on this feature.

Right now, the only hardware environments being measured are our Citrine physical hardware environment and Azure D3v2 instances. However, we are implementing a means for users to contribute and visualize results from other hardware environments for comparison.

Hardware performance measurements must use the specific commit for a round (such as 801ee924 for Round 19) to be comparable, since the test implementations continue to evolve over time.

Because a hardware performance measurement shouldn’t take 4.6 days to complete, we use a subset of the project’s immense number of frameworks when measuring hardware performance. We’ve selected and flagged frameworks that represent the project’s diversity of technology platforms. Any results files that include this subset can be used for measuring hardware environment performance.

The set of TPR-flagged frameworks will evolve over time, especially if we receive further input from the community. Our goal is to constrain a run intended for hardware performance measurement to several hours of execution time rather than several days. As a result, we want to keep the total number of flagged frameworks somewhere between 15 to 25.

You can read more about TPR at the GitHub wiki.

Other Round 19 Updates

Once again, Nate Brady tracked interesting changes since the previous round at the GitHub repository for the project. In summary:

• Updated database platforms: MySQL to v8.0, MongoDB to v4.2, and Postgres to v12.
• Additional database test verifications to ensure implementations are valid.
• Migrated the Citrine hardware environment from CentOS to Ubuntu 18.04.
• Migrated the Azure cloud environment from Ubuntu 16.04 to 18.04 and modified its Terraform script to select a commit recently used in Citrine for easier comparisons.
• Consolidated test requirements on GitHub.
• Clarification of requirements for the World objects used in database tests.
• Load generator update.

Notes

Thanks again to contributors and fans of this project! As always, we really appreciate your continued interest, feedback, and patience!

Round 19 is composed of:

• Run 9a267248-17b0-4080-ac45-a50d25b4fc2a from Azure.
• Run 4c536195-90ff-40b8-8636-a719318a864b from Citrine.

Good fortunes

As the previous round—Round 12—was wrapping up, we were unfortunately rushed as the project’s physical hardware environment was being decommissioned. But good fortune was just around the corner, thanks to the lucky number 13!

New hardware and cloud environments

For Round 13, we have all new test environments, for both physical hardware and the virtualized public cloud.

Microsoft has provided the project with Azure credits, so starting with Round 13, the cloud environment is on Azure D3v2 instances. Previous rounds’ cloud tests were run on AWS.

Meanwhile, ServerCentral has provided the project a trio of physical servers in one of their development lab environments with 10 gigabit Ethernet. Starting with Round 13, the physical hardware environment is composed of a Dell R910 application server (4x 10-Core E7-4850 CPUs) and a Dell R420 database server (2x 4-Core E5-2406 CPUs).

We’d like to extend huge thanks to ServerCentral and Microsoft for generously supporting the project!

We recognize that as a result of these changes, Round 13 is not easy to directly compare to Round 12. Although changing the test environments was not intentional, it was necessary. We believe the results are still as valuable as ever. An upside of this environment diversity is visibility into the ways various frameworks and platforms work with the myriad variables of cores, clock speed, and virtualization technologies. For example, our new physical application server has twice as many HT cores as the previous environment, but the CPUs are older, so there is an interesting balance of higher concurrency but potentially lower throughput. In aggregate, the Round 13 results on physical hardware are generally lower due to the older CPUs, all else being equal.

Many fixes to long-broken tests

Along with the addition of new frameworks, Round 13 also marks a sizeable decrease in the number of existing framework tests that have failed to execute properly in previous rounds. This is largely the result of a considerable community effort over the past few months to identify and fix dozens of frameworks, some of which we haven’t been able to successfully test since 2014.

Continuous benchmarking

Round 13 is the first round conducted with what we’re calling Continuous Benchmarking. Continuous Benchmarking is the notion of setting up the test environment to automatically reset to a clean state, pull the latest from the source repository, prepare the environment, execute the test suite, deliver results, and repeat.

There are many benefits of Continuous Benchmarking. For example:

• At any given time, we can grab the most recent results and mark them as a preview or final for an official Round. This should allow us to accelerate the delivery of Rounds.
• With some additional work, we will be able to capture and share results as they are made available. This should give participants in the project much quicker insight into how their performance tuning efforts are playing out in our test environment. Think of it as continuous integration but for benchmark results. Our long-term goal is to provide a results viewer that plots performance results over time.
• Any changes that break the test environment as a whole or a specific framework’s test implementation should be visible much earlier. Prior to Continuous Benchmarking, breaking changes were often not detected until a preview run.

Microsoft’s ASP.NET Core

We consider ourselves very fortunate that our project has received the attention that it has from the web framework community. It has become a source of a great pride for our team. Among every reaction and piece of feedback we’ve received, our very favorite kind is when a framework maintainer recognizes a performance deficiency highlighted by this project and then works to improve that performance. We love this because we think of it as a small way of improving performance of the whole web, and we are passionate about performance.

Round 13 is especially notable for us because we are honored that Microsoft has made it a priority to improve ASP.NET’s performance in these benchmarks, and in so doing, improve the performance of all applications built on ASP.NET.

Thanks to Microsoft’s herculean performance tuning effort, ASP.NET—in the new cross-platform friendly form of ASP.NET Core—is now a top performer in our Plaintext test, making it among the fastest platforms at the fundamentals of web request routing. The degree of improvement is absolutely astonishing, going from 2,120 requests per second on Mono in Round 11 to 1,822,366 requests per second on ASP.NET Core in Round 13. That’s an approximately 85,900% improvement, and that doesn’t even account for Round 11’s hardware being faster than our new hardware. That is not a typo, it’s 859 times faster! We believe this to be the most significant performance improvement that this project has ever seen.

By delivering cross-platform performance alongside their development toolset, Microsoft has made C# and ASP.NET one of the most interesting web development platforms available. We have a brief message to those developers who have avoided Microsoft’s web stack thinking it’s “slow” or that it’s for Windows only: ASP.NET Core is now wicked sick fast at the fundamentals and is improving in our other tests. Oh, and of course we’re running it on Linux. You may be thinking about the Microsoft of 10 years ago.

The best part, in our opinion, is that Microsoft is making performance a long-term priority. There is room to improve on our other more complex tests such as JSON serialization and Fortunes (which exercises database connectivity, data structures, encoding of unsafe text, and templating). Microsoft is taking on those challenges and will continue to improve the performance of its platform.

Our Plaintext test has historically been a playground for the ultra-fast Netty platform and several lesser-known/exotic platforms. (To be clear, there is nothing wrong with being exotic! We love them too!) Microsoft’s tuning work has brought a mainstream platform into the frontrunners. That achievement stands on its own. We congratulate the Microsoft .NET team for a massive performance improvement and for making ASP.NET Core
a mainstream option that has the performance characteristics of an acutely-tuned fringe platform. It’s like an F1 car that anyone can drive. We should all be so lucky.

How much does your framework choice affect performance? The answer may surprise you.

Authors’ Note: We’re using the word “framework” loosely to refer to platforms, micro-frameworks, and full-stack frameworks. We have our own personal favorites among these frameworks, but we’ve tried our best to give each a fair shot.

Show me the winners!

We know you’re curious (we were too!) so here is a chart of representative results.

Whoa! Netty, Vert.x, and Java servlets are fast, but we were surprised how much faster they are than Ruby, Django, and friends. Before we did the benchmarks, we were guessing there might be a 4x difference. But a 40x difference between Vert.x and Ruby on Rails is staggering. And let us simply draw the curtain of charity over the Cake PHP results.

If these results were surprising to you, too, then read on so we can share our methodology and other test results. Even better, maybe you can spot a place where we mistakenly hobbled a framework and we can improve the tests. We’ve done our best, but we are not experts in most of them so help is welcome!

Motivation

Among the many factors to consider when choosing a web development framework, raw performance is easy to objectively measure. Application performance can be directly mapped to hosting dollars, and for a start-up company in its infancy, hosting costs can be a pain point. Weak performance can also cause premature scale pain, user experience degradation, and associated penalties levied by search engines.

What if building an application on one framework meant that at the very best your hardware is suitable for one tenth as much load as it would be had you chosen a different framework? The differences aren’t always that extreme, but in some cases, they might be. It’s worth knowing what you’re getting into.

Simulating production environments

For this exercise, we aimed to configure every framework according to the best practices for production deployments gleaned from documentation and popular community opinion. Our goal is to approximate a sensible production deployment as accurately as possible. For each framework, we describe the configuration approach we’ve used and cite the sources recommending that configuration.

We want it all to be as transparent as possible, so we have posted our test suites on GitHub.

Results

We ran each test on EC2 and our i7 hardware. See Environment Details below for more information.

JSON serialization test

First up is plain JSON serialization on Amazon EC2 large instances. This is repeated in the introduction, above.

Dedicated hardware

Here is the same test on our Sandy Bridge i7 hardware.

Database access test (single query)

How many requests can be handled per second if each request is fetching a random record from a data store? Starting again with EC2.

Dedicated hardware

Database access test (multiple queries)

The following tests are all run at 256 concurrency and vary the number of database queries per request. The tests are 1, 5, 10, 15, and 20 queries per request. The 1-query samples, leftmost on the line charts, should be similar (within sampling error) of the single-query test above.

Dedicated hardware

How we designed the tests

This exercise aims to provide a “baseline” for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application’s performance can only get worse. We aim to know the upper bound being set on an application’s performance per unit of hardware by each platform and framework.

But we also want to exercise some of the frameworks’ components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it’s a reasonable starting point.

For the data-connected test, we’ve deliberately constructed the tests to avoid any framework-provided caching layer. We want this test to require repeated requests to an external service (MySQL or MongoDB, for example) so that we exercise the framework’s data mapping code. Although we expect that the external service is itself caching the small number of rows our test consumes, the framework is not allowed to avoid the network transmission and data mapping portion of the work.

Not all frameworks provide components for all of the tests. For these situations, we attempted to select a popular best-of-breed option.

Each framework was tested using 2^3 to 2^8 (8, 16, 32, 64, 128, and 256) request concurrency. On EC2, WeigHTTP was configured to use two threads (one per core) and on our i7 hardware, it was configured to use eight threads (one per HT core). For each test, WeigHTTP simulated 100,000 HTTP requests with keep-alives enabled.

For each test, the framework was warmed up by running a full test prior to capturing performance numbers.

Finally, for each framework, we collected the framework’s best performance across the various concurrency levels for plotting as peak bar charts.

We used two machines for all tests, configured in the following roles:

• Application server. This machine is responsible for hosting the web application exclusively. Note, however, that when community best practices specified use of a web server in front of the application container, we had the web server installed on the same machine.
• Load client and database server. This machine is responsible for generating HTTP traffic to the application server using WeigHTTP and also for hosting the database server. In all of our tests, the database server (MySQL or MongoDB) used very little CPU time; and WeigHTTP was not starved of CPU resource. In the database tests, the network was being used to provide result sets to the application server and to provide HTTP responses in the opposite direction. However, even with the quickest frameworks, network utilization was lower in database tests than in the plain JSON tests, so this is unlikely to be a concern.

Ultimately, a three-machine configuration would dismiss the concern of double-duty for the second machine. However, we doubt that the results would be noticeably different.

The Tests

We ran three types of tests. Not all tests were run for all frameworks. See details below.

JSON serialization

For this test, each framework simply responds with the following object, encoded using the framework’s JSON serializer.

{"message" : "Hello, World!"}

With the content type set to application/json. If the framework provides no JSON serializer, a best-of-breed for the platform is selected. For example, on the Java platform, Jackson was used for frameworks that do not provide a serializer.

Database access (single query)

In this test, we use the ORM of choice for each framework to grab one simple object selected at random from a table containing 10,000 rows. We use the same JSON serialization tested earlier to serialize that object as JSON. Caveat: when the data store provides data as JSON in situ (such as with the MongoDB tests), no transcoding is done; the string of JSON is sent as-is.

As with JSON serialization, we’ve selected a best-of-breed ORM when the framework is agnostic. For example, we used Sequelize for the JavaScript MySQL tests.

We tested with MySQL for most frameworks, but where MongoDB is more conventional (as with node.js), we tested that instead or in addition. We also did some spot tests with PostgreSQL but have not yet captured any of those results in this effort. Preliminary results showed RPS performance about 25% lower than with MySQL. Since PostgreSQL is considered favorable from a durability perspective, we plan to include more PostgreSQL testing in the future.

Database access (multiple queries)

This test repeats the work of the single-query test with an adjustable queries-per-request parameter. Tests are run at 5, 10, 15, and 20 queries per request. Each query selects a random row from the same table exercised in the previous test with the resulting array then serialized to JSON as a response.

This test is intended to illustrate how all frameworks inevitably will converge to zero requests per second as the complexity of each request increases. Admittedly, especially at 20 queries per request, this particular test is unnaturally database heavy compared to real-world applications. Only grossly inefficient applications or uncommonly complex requests would make that many database queries per request.

Environment Details

Notes

• For the database tests, any framework with the suffix “raw” in its name is using its platform’s raw database connectivity without an object-relational map (ORM) of any flavor. For example, servlet-raw is using raw JDBC. All frameworks without the “raw” suffix in their name are using either the framework-provided ORM or a best-of-breed for the platform (e.g., ActiveRecord).

Code examples

You can find the full source code for all of the tests on Github. Below are the relevant portions of the code to fetch a configurable number of random database records, serialize the list of records as JSON, and then send the JSON as an HTTP response.

Cake

View on Github

public function index() {
    $query_count = $this->request->query('queries');
    if ($query_count == null) {
        $query_count = 1;
    }
    $arr = array();
    for ($i = 0; $i < $query_count; $i++) { $id = mt_rand(1, 10000); $world = $this->World->find('first', array('conditions' =>
            array('id' => $id)));
        $arr[] = array("id" => $world['World']['id'], "randomNumber" =>
            $world['World']['randomNumber']);
    }
    $this->set('worlds', $arr);
    $this->set('_serialize', array('worlds'));
}
        

Compojure

View on Github

(defn get-world []
  (let [id (inc (rand-int 9999))] ; Num between 1 and 10,000
    (select world
            (fields :id :randomNumber)
            (where {:id id }))))

(defn run-queries [queries]
  (vec ; Return as a vector
    (flatten ; Make it a list of maps
      (take
        queries ; Number of queries to run
        (repeatedly get

Django

View on Github

def db(request):
    queries = int(request.GET.get('queries', 1))
    worlds = []
    for i in range(queries):
        worlds.append(World.objects.get(id=random.randint(1, 10000)))
    return HttpResponse(serializers.serialize("json", worlds), mimetype="application/json")

Express

View on Github

app.get('/mongoose', function(req, res) {
    var queries = req.query.queries || 1,
        worlds = [],
        queryFunctions = [];

    for (var i = 1; i <= queries; i++ ) {
        queryFunctions.push(function(callback) {
            MWorld.findOne({ id: (Math.floor(Math.random() * 10000) + 1 )})
            .exec(function (err, world) {
                worlds.push(world);
                callback(null, 'success');
            });
        });
    }

    async.parallel(queryFunctions, function(err, results) {
        res.send(worlds);
    });
});

Gemini

View on Github

@PathSegment
public boolean db() {
    final Random random = ThreadLocalRandom.current();
    final int queries = context().getInt("queries", 1, 1, 500);
    final World[] worlds = new World[queries];
    for (int i = 0; i < queries; i++) {
        worlds[i] = store.get(World.class, random.nextInt(DB_ROWS) + 1);
    }
    return json(worlds);
}

Grails

View on Github

def db() {
    def random = ThreadLocalRandom.current()
    def queries = params.queries ? params.int('queries') : 1
    def worlds = []

    for (int i = 0; i < queries; i++) {
        worlds.add(World.read(random.nextInt(10000) + 1))
    }

    render worlds as JSON
}

Node.js

View on Github

if (path === '/mongoose') {
    var queries = 1;
    var worlds = [];
    var queryFunctions = [];
    var values = url.parse(req.url, true);

    if (values.query.queries) {
        queries = values.query.queries;
    }
    res.writeHead(200, {'Content-Type': 'application/json; charset=UTF-8'});

    for (var i = 1; i <= queries; i++) {
        queryFunctions.push(function(callback) {
            MWorld.findOne({ id: (Math.floor(Math.random() * 10000) + 1 )})
                .exec(function (err, world) {
                    worlds.push(world);
                    callback(null, 'success');
                });
        });
    }

    async.parallel(queryFunctions, function(err, results) {
        res.end(JSON.stringify(worlds));

PHP (Raw)

View on Github

$query_count = 1;
if (!empty($_GET)) {
    $query_count = $_GET["queries"];
}
$arr = array();
$statement = $pdo->prepare("SELECT * FROM World WHERE id = :id");
for ($i = 0; $i < $query_count; $i++) { $id = mt_rand(1, 10000); $statement->bindValue(':id', $id, PDO::PARAM_INT);
    $statement->execute();
    $row = $statement->fetch(PDO::FETCH_ASSOC);
    $arr[] = array("id" => $id, "randomNumber" => $row['randomNumber']);
}
echo json_encode($arr);

PHP (ORM)

View on Github

$query_count = 1;
if (!empty($_GET)) {
    $query_count = $_GET["queries"];
}
$arr = array();
for ($i = 0; $i < $query_count; $i++) { $id = mt_rand(1, 10000); $world = World::find_by_id($id); $arr[] = $world->to_json();
}
echo json_encode($arr);

	

Play

View on Github

public static Result db(Integer queries) {
    final Random random = ThreadLocalRandom.current();
    final World[] worlds = new World[queries];

    for (int i = 0; i < queries; i++) {
        worlds[i] = World.find.byId((long)(random.nextInt(DB_ROWS) + 1));
    }

    return ok(Json.toJson(worlds));
}

Rails

View on Github

def db
  queries = params[:queries] || 1
  results = []

  (1..queries.to_i).each do
    results << World.find(Random.rand(10000) + 1) end render :json => results

Servlet

View on Github

        res.setHeader(HEADER_CONTENT_TYPE, CONTENT_TYPE_JSON);

final DataSource source = mysqlDataSource;
int count = 1;

try {
    count = Integer.parseInt(req.getParameter("queries"));
} catch (NumberFormatException nfexc) {
    // Handle exception
}

final World[] worlds = new World[count];
final Random random = ThreadLocalRandom.current();

try (Connection conn = source.getConnection()) {
    try (PreparedStatement statement = conn.prepareStatement(
            DB_QUERY,
            ResultSet.TYPE_FORWARD_ONLY,
            ResultSet.CONCUR_READ_ONLY)) {

        for (int i = 0; i < count; i++) {
            final int id = random.nextInt(DB_ROWS) + 1;
            statement.setInt(1, id);

            try (ResultSet results = statement.executeQuery()) {
                if (results.next()) {
                    worlds[i] = new World(id, results.getInt("randomNumber"));
                }
            }
        }
    }
} catch (SQLException sqlex) {
    System.err.println("SQL Exception: " + sqlex);
}

try {
    mapper.writeValue(res.getOutputStream(), worlds);
} catch (IOException ioe) {
    }
	

Sinatra

View on Github

get '/db' do
  queries = params[:queries] || 1
  results = []

  (1..queries.to_i).each do
    results << World.find(Random.rand(10000) + 1)
  end

  results.to_json
end

Spring

View on Github

@RequestMapping(value = "/db")
public Object index(HttpServletRequest request,
                    HttpServletResponse response, Integer queries) {

    if (queries == null) {
        queries = 1;
    }

    final World[] worlds = new World[queries];
    final Random random = ThreadLocalRandom.current();
    final Session session = HibernateUtil.getSessionFactory().openSession();

    for(int i = 0; i < queries; i++) {
        worlds[i] = (World)session.byId(World.class).load(random.nextInt(DB_ROWS) + 1);
    }

    session.close();

    try {
        new MappingJackson2HttpMessageConverter().write(
        worlds, MediaType.APPLICATION_JSON,
        new ServletServerHttpResponse(response));
    } catch (IOException e) {
        // Handle exception
    }

    return null;
}
	

Tapestry

View on Github

StreamResponse onActivate() {
    int queries = 1;
    String qString = this.request.getParameter("queries");

    if (qString != null) {
        queries = Integer.parseInt(qString);
    }

    if (queries <= 0) {
        queries = 1;
    }

    final World[] worlds = new World[queries];
    final Random rand = ThreadLocalRandom.current();

    for (int i = 0; i < queries; i++) {
        worlds[i] = (World)session.get(World.class, new Integer(rand.nextInt(DB_ROWS) + 1));
    }

    String response = "";

    try {
        response = HelloDB.mapper.writeValueAsString(worlds);
    } catch (IOException ex) {
        // Handle exception
    }

    return new TextStreamResponse("application/json", response);
}
	

Vert.x

View on Github

private void handleDb(final HttpServerRequest req) {
    int queriesParam = 1;
    try {
        queriesParam = Integer.parseInt(req.params().get("queries"));
    } catch(Exception e) {
    }
    final DbHandler dbh = new DbHandler(req, queriesParam);
    final Random random = ThreadLocalRandom.current();
    for (int i = 0; i < queriesParam; i++) {
        this.getVertx().eventBus().send(
            "hello.persistor",
            new JsonObject()
                .putString("action", "findone")
                .putString("collection", "world")
                .putObject("matcher", new JsonObject().putNumber("id",
                (random.nextInt(10000) + 1))), dbh);
    }
}

class DbHandler implements Handler<Message<JsonObject>> {
    private final HttpServerRequest req;
    private final int queries;
    private final List<Object> worlds = new CopyOnWriteArrayList<>();

    public DbHandler(HttpServerRequest request, int queriesParam) {
        this.req = request;
        this.queries = queriesParam;
    }

    @Override
    public void handle(Message<JsonObject> reply) {
        final JsonObject body = reply.body;

        if ("ok".equals(body.getString("status"))) {
            this.worlds.add(body.getObject("result"));
        }

        if (this.worlds.size() == this.queries) {
            try {
                final String result = mapper.writeValueAsString(worlds);
                final int contentLength = result
                    .getBytes(StandardCharsets.UTF_8).length;
                this.req.response.putHeader("Content-Type",
                    "application/json; charset=UTF-8");
                this.req.response.putHeader("Content-Length", contentLength);
                this.req.response.write(result);
                this.req.response.end();
            } catch (IOException e) {
                req.response.statusCode = 500;
                req.response.end();
            }
        }
    }
}

Wicket

View on Github

protected ResourceResponse newResourceResponse(Attributes attributes) {
    final int queries = attributes.getRequest().getQueryParameters()
        .getParameterValue("queries").toInt(1);
    final World[] worlds = new World[queries];
    final Random random = ThreadLocalRandom.current();
    final ResourceResponse response = new ResourceResponse();
    response.setContentType("application/json");
    response.setWriteCallback(new WriteCallback() {
        public void writeData(Attributes attributes) {
            final Session session = HibernateUtil.getSessionFactory()
                .openSession();
            for (int i = 0; i < queries; i++) {
                worlds[i] = (World)session.byId(World.class)
                    .load(random.nextInt(DB_ROWS) + 1);
            }
            session.close();
            try {
                attributes.getResponse().write(HelloDbResponse.mapper
                    .writeValueAsString(worlds));
            } catch (IOException ex) {
            }
        }
    });
    return response;
}

Expected questions

We expect that you might have a bunch of questions. Here are some that we’re anticipating. But please contact us if you have a question we’re not dealing with here or just want to tell us we’re doing it wrong.

1. “You configured framework x incorrectly, and that explains the numbers you’re seeing.” Whoops! Please let us know how we can fix it, or submit a Github pull request, so we can get it right.
2. “Why WeigHTTP?” Although many web performance tests use ApacheBench from Apache to generate HTTP requests, we have opted to use WeigHTTP from the LigHTTP team. ApacheBench remains a single-threaded tool, meaning that for higher-performance test scenarios, ApacheBench itself is a limiting factor. WeigHTTP is essentially a multithreaded clone of ApacheBench. If you have a recommendation for an even better benchmarking tool, please let us know.
3. “Doesn’t benchmarking on Amazon EC2 invalidate the results?” Our opinion is that doing so confirms precisely what we’re trying to test: performance of web applications within realistic production environments. Selecting EC2 as a platform also allows the tests to be readily verified by anyone interested in doing so. However, we’ve also executed tests on our Core i7 (Sandy Bridge) workstations running Ubuntu 12.04 as a non-virtualized sanity check. Doing so confirmed our suspicion that the ranked order and relative performance across frameworks is mostly consistent between EC2 and physical hardware. That is, while the EC2 instances were slower than the physical hardware, they were slower by roughly the same proportion across the spectrum of frameworks.
4. “Why include this Gemini framework I’ve never heard of?” We have included our in-house Java web framework, Gemini, in our tests. We’ve done so because it’s of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we’re proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.
5. “Why is JRuby performance all over the map?” During the evolution of this project, in some test runs, JRuby would slighly edge out traditional Ruby, and in some cases—with the same test code—the opposite would be true. We also don’t have an explanation for the weak performance of Sinatra on JRuby, which is no better than Rails. Ultimately we’re not sure about the discrepancy. Hopefully an expert in JRuby can help us here.
6. “Framework X has in-memory caching, why don’t you use that?” In-memory caching, as provided by Gemini and some other frameworks, yields higher performance than repeatedly hitting a database, but isn’t available in all frameworks, so we omitted in-memory caching from these tests.
7. “What about other caching approaches, then?” Remote-memory or near-memory caching, as provided by Memcached and similar solutions, also improves performance and we would like to conduct future tests simulating a more expensive query operation versus Memcached. However, curiously, in spot tests, some frameworks paired with Memcached were conspicuously slower than other frameworks directly querying the authoritative MySQL database (recognizing, of course, that MySQL had its entire data-set in its own memory cache). For simple “get row ID n” and “get all rows” style fetches, a fast framework paired with MySQL may be faster and easier to work with versus a slow framework paired with Memcached.
8. “Do all the database tests use connection pooling?” Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling.
9. “What is Resin? Why aren’t you using Tomcat for the Java frameworks?” Resin is a Java application server. The GPL version that we used for our tests is a relatively lightweight Servlet container. Although we recommend Caucho Resin for Java deployments, in our tests, we found Tomcat to be easier to configure. We ultimately dropped Tomcat from our tests because Resin was slightly faster across all frameworks.
10. “Why don’t you test framework X?” We’d love to, if we can find the time. Even better, craft the test yourself and submit a Github pull request so we can get it in there faster!
11. “Why doesn’t your test include more substantial algorithmic work, or building an HTML response with a server-side template?” Great suggestion. We hope to in the future!
12. “Why are you using a (slightly) old version of framework X?” It’s nothing personal! We tried to keep everything fully up-to-date, but with so many frameworks it became a never-ending game of whack-a-mole. If you think an update will affect the results, please let us know (or submit a Github pull request) and we’ll get it updated!

Conclusion

Let go of your technology prejudices.

We think it is important to know about as many good tools as possible to help make the best choices you can. Hopefully we’ve helped with one aspect of that.

Thanks for sticking with us through all of this! We had fun putting these tests together, and experienced some genuine surprises with the results. Hopefully others find it interesting too. Please let us know what you think or submit Github pull requests to help us out.

About TechEmpower

We provide web and mobile application development services and are passionate about application performance. Read more about what we do.