Node.js Build scalable architecture with plugins

A dream architecture of any server engineer is a set of highly modularized and reusable components. However, that kind of architecture proves to be difficult to achieve. This is especially true as your code base keeps growing. One trick that might help scaling your code is separating your reusable components into plugins. In this short blog post I want to go over the benefits and reasons for using this approach.

What is a plugin?

A plugin is a component that extends the functionality of the main application in an extensible and reusable way. One of the reasons Node.js is so popular is not simply because of its superior architecture, but also because of its large number of plugins as packages, created by the community, which provide us with almost any functionality we might need.

Distributing the plugin

Public plugins

As mentioned before, a vast majority of plugins are distributed as npm packages that can be installed into your project’s node_modules folder. All of the public plugins are distributed through npm and are available for use by anybody. One of the most prominent examples of such plugin is express .

Private plugins

However, plugins do not necessarily need to be public. Plugins can also be written for a private use only, within the organization, or even a single project. There are different ways to distribute and share private plugins:

Version control system – you can simply add your plugin to the version control used by your team.
Private npm package – a lot of organizations use npm to package their plugins and share it within the organization only.

While internal plugins may use the dependencies of of the main application, it can also be beneficial for the plugin to have its own dependency graph and package.json. Here’s a sample diagram:

project
|_package.json
|_node_modules
     |_pluginA
         |_package.json
     |_pluginB
         |_package.json

As you can see, we have three package.json files in our project: one for the main application and two for each plugin in the node_modules folder. Here are three reasons why you might want go with this approach:

NPM – We can leverage npm to distribute our plugins as private packages.
Dependency graph – These plugins have their own dependency graphs which makes it easier to maintain project’s dependencies and resolve conflicts.
Easier to import – It’s much easier to import these plugins. Instead of using relative paths (require('../../utils/moduleA')), we only need to provide the name of the plugin(require('pluginA')).

Why organize your code into plugins?

Better structure

So why use plugins at all? In addition to the previous considerations, one of the main reasons to separate part of your application’s functionality into a plugin is the overall architecture. By separating your code into plugins, you separate your code into units that are both reusable and loosely coupled. It forces you to consider encapsulation by thinking about which parts of the plugin should be exposed to the main app. Overall, it makes your application more scalable.

Serverless architecture

Serverless architecture is arguably the future of the web development. While this kind of architecture provides better scalability over the traditional servers, there’s this problem of sharing common components between serverless functions.

A function in a serverless architecture is usually a relatively small snippet of code that performs certain computation or task. It’s usually setup as a stand-alone microservice and by default does not share code with other functions in the project.

As mentioned before, since each function is a standalone service, it can be bothersome to share common code between the functions. The reason is because when deployed, each function runs on a separate environment.

This is where packaging your plugins and distributing them as npm modules can help. All you need to do is publish your plugin as a private package and install it in your functions as needed.

Writing a plugin

Let’s see how we could write a plugin that extends the functionality of the simple serverless function. Imagine this kind of project structure.

serverlessF
 |_node_modules
	|_pluginA
		|_index.js
		|_package.json
 |_fetch.js
 |_app.js
 |_package.json

utils
 |_pluginA
	|_index.js
	|_package.json

A few things to note here:

serverlessF folder contains our serverless function. It’s a simple Node.js application.
utils folder is meant for code that is shared between multiple functions in our project. It consists of different plugins, each with its own package.json.
pluginA is published as a private package and installed by our serverless function.

Here’s a sample code for a simple plugin that validates a third party token:

1
2
3
4
5
6
7
8
9


const fetch = module.parent.require('./fetch');

const validateRequest = async (req) =&gt; {
  const { token } = req.body;
  const result = await fetch(token);
  // further validate token
};

module.exports = validateRequest;

Our plugin exposes one function for validating the token. One interesting trick here is the first line. If you look at the diagram above you will notice that our lambda function contains fetch.js which for our purpose is used to make various fetch requests to the third party API. Using module.parent.require('./fetch') we were able to use that module within our plugin by imitating the require call from the parent application where the plugin is installed.

Although using module.parent.require might seem like a nifty trick, a lot of times it’s advised against relying on it. The reasons is that it creates tight coupling between our serverless function and the plugin. If we were to move fetch.js to a different subfolder, our plugin would break. A better approach would be to use dependency injection to provide our plugin with an instance of the fetch service:

1
2
3
4
5
6
7


const validateRequest = async (req, fetch) =&gt; {
  const { token } = req.body;
  const result = await fetch(token);
  // further validate token
};

module.exports = validateRequest;

NOTE: When writing plugins, its especially important to avoid hardwiring dependencies and creating tight coupling, otherwise the purpose of the plugin is defeated. You can read more about module wiring and dependencies here .

Conclusion

As a small disclaimer, you don’t have to separate every part of your application into a plugin as that might add more complexity than necessary. However, if you have a big dependency graph and/or a set of cohesive modules, it might be worth splitting some of them into plugins. Furthermore, packaging your code into plugins can be a good way to share common code between serverless functions.

If you’d like to get more web development, React and TypeScript tips consider following me on Twitter, where I share things as I learn them.
Happy coding!