Behind the Scenes of ExpressJS: A Deep Dive into Nodejs’ Most Popular Library
Express is one of the most used Nodejs libraries and it is really important to really know this library and be more than just a user. It is not a complex library and in this article, we want to get some insights into what is going on internally. This might help us better understand when there is a problem in our express app.
What You Will Read In This Article
This article assumes that you already worked a little bit with ExpressJS and you want to dig a bit more into ExpressJS.
First, we check out Why Express Is Important and the pros and cons.
Then we are going to check out the source code of ExpressJS and point out important parts:
- Create Express Application
- Adding Middleware
- Handle Request
- Next Function
Why Express?
Express is still one of the most used backend frameworks in the world so it is worth the shot to see the pros and cons and then some insights into the internal implementations.
I have been working with ExpressJS for the last 5 years and based on my personal experience these are the pros and cons.
Pros
- Widely used: as you can see in this visual report there is still a good demand for job opportunities.
- Community and a lot of integrated tools: Since so many people are using ExpressJS you can find a lot of libraries solving the problems around ExpressJS.
- Simplicity: It takes a few minutes for a junior developer to get how to create a simple app with ExpressJS.
- Middleware: Nowadays the idea of middleware is implemented in most frameworks which is a good thing and it solves some reoccurring problems in applications.
Cons
- Performance: You can see in the benchmark that it is not the fastest HTTP library in Nodejs and compared to something like Fastify it is slow!
- Security: It does not offer some of the security checks needed capabilities or validations by default so you need to implement them yourself or use a third-party library.
Let’s See How Express Handles the Request!
Clone The repository of the Express
Use one of these commands:
git clone https://github.com/expressjs/express.gitgit clone git@github.com:expressjs/express.gitIf you do not want to clone the repository that is totally OK I put important codes in here.
What happens when we create an Express app(create application Function)?
We usually create our Express instance like this:
const app = require('express')();We are running the default exported function. We can find this function in lib/express.js
var bodyParser = require('body-parser');
var EventEmitter = require('events').EventEmitter;
var mixin = require('merge-descriptors');
var proto = require('./application');
var Route = require('./router/route');
var Router = require('./router');
var req = require('./request');
var res = require('./response');
exports.json = bodyParser.json;
exports.text = bodyParser.text;
exports.raw = bodyParser.raw;
exports.urlencoded = bodyParser.urlencoded;
exports = module.exports = createApplication;
function createApplication() {
var app = function(req, res, next) {
app.handle(req, res, next);
};
mixin(app, EventEmitter.prototype, false);
mixin(app, proto, false);
// expose the prototype that will get set on requests
app.request = Object.create(req, {
app: { configurable: true, enumerable: true, writable: true, value: app }
})
// expose the prototype that will get set on responses
app.response = Object.create(res, {
app: { configurable: true, enumerable: true, writable: true, value: app }
})
app.init();
return app;
}So the default function is createApplication .
Let’s see what is going on along these lines:
body-parser library is imported and these four methods have been used: json , text urlencodedand raw . So each time you are calling res.json({ someData: true }) in your Express Application, you are actually using body-parser library. (If you are curious how this library parses your request body check this file out)
Inside createApplication function there is a function called mixin which is the merge-descriptors library. This function simply adds the methods of the second argument to the first object which is passed.
const mixin = require('merge-describpto')
var thing = {
get name() {
return 'jon'
}
};
var animal = {
};
mixin(animal, thing);
animal.name === 'jon'Two objects are mixed with our express instance. The first one is event emitter which is for having event emitting behavior(we want to listen to events like errors and also we want to emit some events for letting know the listeners that something has happened) and the second one is proto which is this file in the repository: lib/application.js
In the third part of the createApplication it assigns request and response objects from these two files:
lib/request.js and lib/response.js .
Another thing you can find here is the listen method:
app.listen = function listen() {
var server = http.createServer(this);
return server.listen.apply(server, arguments);
};Here you can see that this is where express passes the object that handles the requests and does all the logic to the Nodejs HTTP core library.
Express Router: Adding Middlewares
One of the important methods we use on Express instance is use function, when we want to register a middleware on our application(lib/router/index.js ):
proto.use = function use(fn) {
var offset = 0;
var path = '/';
// default path to '/'
// disambiguate router.use([fn])
if (typeof fn !== 'function') {
var arg = fn;
while (Array.isArray(arg) && arg.length !== 0) {
arg = arg[0];
}
// first arg is the path
if (typeof arg !== 'function') {
offset = 1;
path = fn;
}
}
var callbacks = flatten(slice.call(arguments, offset));
if (callbacks.length === 0) {
throw new TypeError('Router.use() requires a middleware function')
}
for (var i = 0; i < callbacks.length; i++) {
var fn = callbacks[i];
if (typeof fn !== 'function') {
throw new TypeError('Router.use() requires a middleware function but got a ' + gettype(fn))
}
// add the middleware
debug('use %o %s', path, fn.name || '<anonymous>')
var layer = new Layer(path, {
sensitive: this.caseSensitive,
strict: false,
end: false
}, fn);
layer.route = undefined;
this.stack.push(layer);
}
return this;
};In the first part of this function, it checks if only one middleware is passed or if it is an array. As you know we are allowed to use it like this:
app.use(firstMiddleware, secondMiddleware);
app.use([firstMiddleware, secondMiddleware]);Then it checks and returns an error if no middleware is passed. I get this error if I use something like this:
app.use(); // or app.use([])
As you can see this is the same error we are getting: requires a middleware function .
In the next part, you see that it flattens the callbacks. This is also important. We are allowed to pass middlewares even in nested arrays. An example would be something like this:
const app = require('./lib/express')();
const first = (req, res, next) => {console.log('first');next();}
const second = (req, res, next) => {console.log('second');next();}
const third = (req, res, next) => {console.log('third');res.send('text');}
app.use([first, [second, third]]);So use function needs to flatten these middlewares to one array like this:
[1, [2, 3]] => Flatten => [1, 2, 3]This is what this flatten function is doing. Now that we have an array of middlewares we need to loop over them and add them to keep them somewhere. This is the next step:
var layer = new Layer(path, {
sensitive: this.caseSensitive,
strict: false,
end: false
}, fn);
layer.route = undefined;
this.stack.push(layer);The most important thing in this part is that we are adding these middlewares(in a slightly different shape) to an object or array that implements a push function, called stack this is where Express keeps the middlewares and executes the function when needed.Layer is a simple constructor function that keeps the data of middleware in a different shape:
You can find Layer here: lib/router/layer.js . As you can see it is just some assignment and nothing really important happens.
function Layer(path, options, fn) {
if (!(this instanceof Layer)) {
return new Layer(path, options, fn);
}
debug('new %o', path)
var opts = options || {};
this.handle = fn;
this.name = fn.name || '<anonymous>';
this.params = undefined;
this.path = undefined;
this.regexp = pathRegexp(path, this.keys = [], opts);
// set fast path flags
this.regexp.fast_star = path === '*'
this.regexp.fast_slash = path === '/' && opts.end === false
}Express Router: Handle Request and Next Function
We saw that we have a function called handle.
Now we want to see how a request is handled. It calls the handle function in this file lib/router/index.js . proton.handle :
I intentionally removed some parts of the handle function so we can focus on important parts:
proto.handle = function handle(req, res, out) {
var self = this;
var idx = 0;
var sync = 0;
// middleware and routes
var stack = self.stack;
// setup next layer
req.next = next;
next();
// ... here is the next function implementation
};idx is for keeping track of the current index in the array of middlewares(stack). It calls the next function. Let’s see the implementation of the next function. (Also here, some parts are deleted to focus on the main parts)
function next(err) {
// no more matching layers
if (idx >= stack.length) {
setImmediate(done, layerError);
return;
}
// find next matching layer
var layer;
var match;
var route;
while (match !== true && idx < stack.length) {
layer = stack[idx++];
// Match the current path and middlewares
match = matchLayer(layer, path);
route = layer.route;
if (match !== true) {
continue;
}
if (layerError) {
// routes do not match with a pending error
match = false;
continue;
}
var method = req.method;
var has_method = route._handles_method(method);
}
// no match
if (match !== true) {
return done(layerError);
}
// store route for dispatch on change
if (route) {
req.route = route;
}
// this should be done for the layer
self.process_params(layer, paramcalled, req, res, function (err) {
if (err) {
next(layerError || err)
} else if (route) {
layer.handle_request(req, res, next)
} else {
trim_prefix(layer, layerError, layerPath, path)
}
sync = 0
});
}The first thing it checks is if there are more middlewares that should be checked or not: if (idx >= stack.length) so it returns if there is nothing more to check.
The next step is to check which layers match the current request. You see a while loop that runs until it finds a match middleware. In the while loop you can see a check for finding the match: match = matchLayer(layer, path);.
The last part of the implementation is calling self.process_params for some checks and if there is no error it finally calls layer.handle_request which is the function (req, res, next)=>{...} that you passed to the matching middleware. If that middleware returns the response then the process with this request is done otherwise, it calls the next function and it goes to the same process again with one simple difference in the value idx , so it finds the next match in the array of middlewares.
Summary
Although we neglected so many details in ExpressJS implementation we have a better understanding of where express keeps the middlewares and how it handles each request.
The first thing we checked was the creation of an Express instance and how Express adds different functionalities to this object and finally serves the HTTP server with listen function.
Then we saw that by calling use or method functions on an Express instance we can add these middlewares to an object called stack .
For handling a request Express calls the handle function and it calls the next function after some checking and changes in the parameters and next function has access to the stack object that the middlewares are in there, then it loops over the middlewares and if it finds a match it calls the middleware.
Another related Article from me that might be interesting for you:
Build your First Node.js Web Framework Step by Step
