Announcing Meteor 1.5

Dynamic import(…), exact code splitting, immutable module caching, and bundle analysis tools

Today we’re thrilled to announce the release of Meteor 1.5, which has been more than four months in the making, with significant improvements around JavaScript bundle sizes and page load performance.

How to upgrade

To update any existing application to Meteor 1.5, simply run meteor update in the application directory. To install Meteor from scratch, consult the installation instructions on our website.

As a reminder, make sure your application’s .meteor directory is committed to your version-control system (e.g. Git, Subversion, etc.) so it’s easy to roll back if you encounter problems during any upgrade.

A little Meteor history

When Meteor was first released in late 2011, most websites were still submitting forms and reloading themselves to display changes. Using the recently-standardized XMLHttpRequest API to fetch dynamic content was all the rage, and jQuery was hugely popular. At the time, I was working for Quora, whose live-updating questions and answers, real-time notifications, and hyperactive news feed felt like a significant competitive advantage.

In that context, Meteor’s vision was revolutionary: What if your data could be refreshed automatically whenever anyone changed it, and the user interface of your application would update accordingly? What if you sent only data over the network, and let the client render it? What if reactivity was the default, not an occasional trick? What if you didn’t have to be Quora, or Asana, or even hire a team of engineers, to build an app like that? What if it was all open source?

In order to implement that vision, Meteor made bold, opinionated tradeoffs. Every Meteor application would send and receive its data over a WebSocket, at a time when native WebSocket support was far from universal. We embraced MongoDB because it provided a real-time log of changes that could power our data system. We added low-level support for coroutines (fibers) to Node in hopes of simplifying our asynchronous APIs, before the Promise abstraction was standardized. We built our own packaging system before npm became the way everyone shares their JavaScript code.

At the time, those tradeoffs were right for many of our developers. As time has passed, however, new technologies and best practices have emerged from the wider ecosystem, creating opportunities to revisit some of the choices we previously made. In many ways, that realignment has been the narrative behind every major Meteor release since I joined the company in September 2014.

What Meteor is (and isn’t)

In its essence, Meteor is a tool for building rich, client-side applications that remain up-to-date even as multiple clients interact with the application simultaneously.

Historically, this emphasis on long-running single-page applications has prevented Meteor from being a very good tool for building lightning-fast landing pages or web crawler-friendly static websites. Rather than optimizing for initial page load times, Meteor prioritizes the needs of users who load the application once and then interact with it for an extended period of time. Arguably, the initial JavaScript bundle can take more time to download if the code it contains enables near-instantaneous data updates, makes the application more predictable, and saves you from duplicating the work of other users because you didn’t see it soon enough.

Still, first impressions matter, and a large Meteor app can take quite a bit of time to load, especially over a slow network, if the initial JavaScript bundle is not already cached by the browser.

It’s time we revisited these particular tradeoffs, and Meteor 1.5 empowers developers to improve page load performance significantly, without compromising any of Meteor’s traditional strengths.

Code splitting

Meteor is hardly the first web framework to provide a means of “code splitting,” or delivering JavaScript on demand, in smaller fragments, rather than in one monolithic initial bundle.

My earliest experience with code splitting was in the summer of 2006, as an intern at Meebo, the web-based instant messaging startup. I vividly remember the full-time engineers attempting to use dojo.require and dojo.provide to defer fetching code that wasn’t necessary to display the buddy list. Ultimately, the performance benefits were not dramatic enough to justify maintaining the system, and the project was put on hold. Maybe they eventually got back to it, but my point is that code splitting is a notoriously tricky, relatively old idea, and by no means a silver bullet.

In the decade since then, JavaScript has changed in ways that make code splitting much, much easier. Our JavaScript engines are orders of magnitude faster, our browsers are smarter, we have a much better idea how modules ought to work, and just recently a new language feature was proposed—dynamic import(…)—that promises a standard way to request additional code from the server.

Meteor 1.5 contains many improvements, but the one that overshadows all the rest is a completely new implementation of the proposed dynamic import(…) syntax, backed by a sophisticated caching and module fetching system that makes the most of Meteor’s unique advantages.

Dynamic import(…) in Meteor

First and most importantly, Meteor’s implementation of dynamic import(…) requires no special configuration whatsoever. If you’ve used dynamic import(…) already in another framework, chances are it was based on the implementation in Webpack 2, perhaps hidden behind a layer of abstraction. If you dug into that abstraction, you would find a configuration surface area that—let’s just say—would make you appreciate the abstraction. To get a sense for what I’m talking about, see this in-depth discussion of how to optimize your code splitting in Webpack 2, which culminates in a 95-line, carefully considered configuration file that enables magic like the BundleAnalyzerPlugin and the CommonsChunkPlugin, neither of which is necessary nor makes any sense in Meteor.

In Meteor 1.5, if you wish to defer loading a JavaScript module (and all of its dependencies) until later, you simply turn

import { a, b as c } from "./some/module";
doSomething(a, c)

into

import("./some/module").then(({ a, b: c }) => {
  doSomething(a, c);
});

This API should be familiar if you’ve used Promises before, because import(“./some/module”) returns a Promise for the exports of the requested module.

If you prefer, you even can await the result using an async function:

async start() {
  const { a, b: c } = await import("./some/module");
  return doSomething(a, c);
}

Either way, since you’re now using dynamic import(…) instead of a traditional import declaration or require call, ./some/module will be removed from your initial JavaScript bundle, along with any modules it depends on that would not otherwise be bundled. End of story.

Bundle analysis tools

If you care at all about your bundle size, then you must also care about measuring the impact of your improvements.

In tandem with Meteor 1.5, we’ve released a package called bundle-visualizer that can help you understand the breakdown of your Meteor packages and the JavaScript modules they contain, in proportion to the total size of your initial bundle.

To use this package, simply run

cd path/to/your/meteor/1.5/app
meteor add bundle-visualizer
meteor run --production

then open your app in a web browser. As you move your cursor over the sunburst chart, you can easily see which packages, directories, and modules are most responsible for the size of your bundle:

You can see at a glance that this to-dos application is suffering—rather massively—from the accidental inclusion of the faker npm package in production. Importing this library dynamically (in addition to removing the unneeded locales) would cut the bundle size almost in half!

I find it particularly delightful that the chart itself is powered by a dynamically-imported d3 library. In fact, if the visualizer was not imported dynamically, then it would skew the very data it is helping you to visualize, since the bundle-visualizer package depends on more than 100 kB of node_modules.

Note that the --production flag is important because measuring bundle sizes based on unminified source code can be extremely misleading—not to mention disheartening, since a typical Meteor app contains megabytes of unminified code, comments, and whitespace.

Please remember to run meteor remove bundle-visualizer before actually deploying your app to production, since you probably don’t want the visualizer to be visible to your users. On that note, if you have ideas for making the visualizer easier to use, or more useful, you can find its source code here. We look forward to your feedback and pull requests.

Exact code splitting

How does Meteor’s implementation of dynamic import(…) differ from other implementations, like the one in Webpack 2?

Webpack 2 uses your dynamic import(…) calls to pre-build multiple bundles of dynamic modules. There are a lot of interesting details, but the short version is: when you perform a dynamic import(…), one or more of these bundles of modules will be downloaded behind the scenes, containing the module you requested, all of its dependencies, and any other modules that also happen to be in that bundle.

This strategy is appealing because pre-built bundles can be served from a CDN, with HTTP caching, using relatively few HTTP requests. However, bundling also has drawbacks:

1. Cached bundles are frequently invalidated. When any single module in a bundle changes, the whole bundle has to be downloaded again.
2. Bundles overlap. You may end up downloading the same module multiple times in different bundles, because it happens to be a dependency of more than one import(…)ed module. To eliminate any overlap between bundles, in theory, you would have to generate enough different bundles to account for every possible ordering of dynamic import(…) calls, but then you would have so many bundles that choosing between them would be tricky, and the benefits of caching would disappear. In practice, Webpack 2 tolerates the overlap.
3. Effective bundling requires manual configuration. Webpack allows you to configure a “vendor” bundle of infrequently changing code, and the CommonsChunkPlugin moves shared dependencies into a separate bundle, but neither of those techniques is fully automatic, and you should probably still inspect the shared bundles to make sure they don't contain too many modules you don’t need.

By contrast, in the Meteor 1.5 implementation of dynamic import(…), the client has perfect information about which modules were in the initial bundle, which modules are in the browser’s persistent cache, and which modules still need to be fetched. There is never any overlap between requests made by a single client, nor will there be any unneeded modules in the response from the server. You might call this strategy exact code splitting, to differentiate it from bundling.

To avoid the overhead of multiple parallel HTTP requests, Meteor currently fetches dynamic modules using a WebSocket (remember that every Meteor client already has a dedicated socket open to the server). This strategy not only eliminates much of the latency of opening and closing HTTP requests, but also allows for smaller total response sizes, since a single large response tends to compress better than many smaller independent responses, thanks to the way the gzip/deflate algorithms work.

Immutable module caching

The initial JavaScript bundle includes hashes for all available dynamic modules, so the client never has to ask the server if it should use a cached version of a module. If that module’s hash is found in the IndexedDB-based browser cache, then the client can safely use the cached version without any confirmation from the server, and the same version of that module never needs to be downloaded again by the same client. For example, when you update to a totally new version of React, clients who previously visited your site will download only those modules that changed (or were newly introduced) since the previous version. This caching system has all the benefits of immutable caching, if you’ve been hearing about that.

In this implementation, the server doesn’t have to understand the module dependency graph at all, and doesn’t need to remember what modules have already been sent to the client, but merely responds with whatever modules the client says it needs. And because the server is completely stateless, scaling your app to multiple servers/containers can be fully automatic.

To make developing with dynamic modules more predictable and representative of the experience of new visitors, immutable caching is enabled only in production. In other words, don’t be surprised that fetching your dynamic modules in development takes the same amount of time regardless of how many times you’ve fetched them before. To see the impact of the caching, you’ll have to either deploy your app to production, or run it in simulated production mode via meteor run --production.

A word of caution

As I learned during that internship in the summer of 2006, the benefits of code splitting are by no means guaranteed. As you introduce dynamic import(…)s into your application, you should always be measuring the impact, not just in terms of the size of your initial JavaScript bundle, but also the time spent waiting for dynamic modules to arrive.

Generally speaking, if you use the bundle-visualizer to identify large module trees that are not necessary during page load, then it’s probably a good idea to import those modules dynamically. On the other hand, if you take a library that’s critical to the startup of your application and blindly begin importing it dynamically, then your bundle may shrink, but you will probably increase the total time it takes for your application to become ready to use.

What’s next

This implementation of dynamic import(…) is production-ready, but that is not to say there’s nothing left to improve. For example, Meteor could potentially avoid the pitfalls I mentioned in the previous section by detecting when a dynamic module really ought to be in the initial bundle instead, given how soon you typically end up asking for it. As another idea, for some applications, it might make more sense to load dynamic modules via HTTP rather than using a WebSocket. We look forward to your feedback, and we are excited to continue improving this system to suit your needs.

The plan for Meteor 1.6 is simple: upgrade Meteor from Node 4 to Node 6, now that Node 4 no longer enjoys long-term support (LTS), and Node 6 is the official stable version. Much of this work has already happened in parallel with Meteor 1.5, so we anticipate a much shorter release cycle for Meteor 1.6. Please see this pull request if you want to get involved, as your help will surely speed that effort along.

This blog post is by no means our final word on everything new in Meteor 1.5, so stay tuned for follow-up posts in the coming days and weeks. In fact, if you have positive (or at least interesting) experiences with Meteor 1.5 in your own projects, and you have time to contribute an article, we would be more than happy to publish and promote it for you. Just let us know!

Final notes

We recommend reading through the release notes to avoid surprises when updating your apps, and please remember to commit your .meteor directory to version control, just in case you need to roll back to a previous Meteor version. With those recommendations in mind, you should now be ready to meteor update to Meteor 1.5!