Mean

MEAN (Angular, Node) for Django programmers (part 2)

Continuing where I left off in part 1, a list of things I had to struggled with when I switched from Django to MEAN.

Should I embrace Mongo or stick with Postgre/MySQL?

There's no right answer here, so I'll speak from experience. Thinking relational is what I'm used to and it's not trivial to make the shift to schemas that may actually encourage data duplication. At first I decided to stick with MySQL but I quickly switched to Mongo for a few reasons:

  1. There is no free SQL hosting service I felt like using. Heroku's free Postgre tier has a 4 hour monthly downtime which I couldn't stomach
  2. Clearly SQL is a road less traveled with Node/Express. There's a ton of documentation on Mongo integration and defacto standard tools to wrap it (mongoose, for example)
  3. I really wanted to put all the stuff I read about the pitfalls/benefits of document stores to practice. I knew only getting my hands dirty would illustrate the document paradigm for better and worse

To be honest, at first I used Mongo as a relational datastore - I simply converted tables to documents, with references between them. This was done under the incorrect understanding that embedded/sub documents declared with mongoose create seperate documents under the hood anyway. However, once I realized this wasn't the case I quickly switched the schema to hold everything relevant under a single document and things flew (code complexity wise, not performance wise). I had a much easier time accessing my data because the sort of queries I was doing just clicked with how the schema was laid out, and that is how everyone says you should base your schema design anyway.

Is there a place I can deploy my projects for free?

Here's what I use for my project:

  • 1 Heroku dyno
  • MongoLabs: I didn't use the Heroku addon rather signed up directly
  • NewRelic addon: Checks uptime and keeps the dyno warmed up so that it won't take a few seconds to spin up if there's no traffic to it
  • Logentries addon: Aggregates all logs into something that's easy to search and traverse. Free tier has acceptable log retention

How do I choose which tests to run with Mocha and Webstorm?

Mocha supports selecting which tests to run by running it as:

mocha -g <test name>  

Since this is grep-like, you can even pass it something partial - anything in the test name or description. This holds true to running Mocha tests from within Webstorm. Go to Run | Edit Configurations. Then in Extra Mocha options enter -g "" (for example: -g "Patients").

Debug configuration

Modules/helpers I should use from day 1?

There are a billion really popular modules and that makes it difficult to decide which should be used early on. As you gain more and more experience you'll probably have a toolbelt of your go to modules, but choosing incorrectly at an early stage may sometimes mean you'll have to refactor a lot of ugly code that. Here are a few modules I think even test projects should include.

nodemon

A really useful tool to reload your Node service every time you change your code. Since I separate client from server code, I don't want the server to reload every time I change some frontend stuff, so I only instruct it to watch the /server the directory as such:

nodemon server/src/server.js --watch server  

Where server/src/server.js is main Node module and server is the directory containg server code. Note that by entering:

rs  

Into the console, you can force restart the server.

bower

Instead of pulling in frontend JavaScript modules manually (and managing their depenedencies), use bower to handle that. There are enough tutorials out there to explain how to work with bower, but note that you'll have to call bower install on postinstall to get Heroku to install the bower modules.

underscore

I was rather shocked just how inelegant JavaScript is compared to Python. Underscore helps that a bit by offering basic, common functions you would have had to implement yourself. Things like finding in an array, sort'ing, map'ing, etc.

Some promise module (Q / bluebird)

In most Node examples you will find that asynchronous callbacks are passed by supplying an anonymous function:

someAsyncCall(arg1, arg2, function(result) {  
    // do something, sometimes call another async call
}

At first this will seem rather neat, but you will then find yourself struggling with:

  • Nested calls, making the code unreadable
  • Parallelizing several async calls
  • Error handling

You can use async for this, but there's a much more elegant solution which is Promises. If you've ever working with Python Twisted, they are equivalent to a deferred. Promises require quite a bit of practice and the syntax may be a bit odd initially but you're much better off learning this valuable tool before you write the bulk of your code than after.

How can I run Node locally and still be able to receive Webhook calls?

Lets say your backend code calls some service which needs some time to process the request. More often than not, the reply will be called via a Webhook (that is, usually POST that the service calls on your server in response to your request). This means you need to export an endpoint accessible from the internet. If you're a sane person, you will develop and deploy your code locally until it's ready for staging an production - which means you're behind a NAT ("firewall") of some sort.

You could go the port forwarding route but that's tedious and not always possible. Instead, I highly recommend using localtunnel.me. This awesome free service runs an agent on your machine which connects to the localtunnel.me servers and receives a unique, public URL (e.g. eran.localtunnel.me). Any request to this URL made from anywhere will be forwarded to the agent running on your machine.

When you run the agent, you tell it two things:

  • The name of the URL you want (e.g. eran)
  • The port to which to forward all requests the agent receives (namely, the port you run Node on)
lt --port 8080 --subdomain eran  

In this case, I am asking for eran.localtunnel.me and running my Node application on port 8080. If a server executes a webhook and POSTs to eran.localtunnel.me/some/sub/path it will effectively hit my Node application at /some/sub/path. Very cool.

How can I test endpoints that use an API, without hitting an external server?

Simply use nock. Nock lets you override outgoing requests (by instrumenting Node) so that you can both check validity of the request and tailor a response. Check out my payments tests and look for nock.

I use a service which produces government approved invoices called greeninvoice. In these tests, I check that the Node application crafts the correct request towards greeninvoice according to payment method (cash, cheque, transfer). Here's an excerpt of me telling nock to catch any outgoing requests to greeninvoice:

// expect invoice and check all parameters, including cash
nock('https://api.greeninvoice.co.il')  
    .post('/api/documents/add')
    .reply(200, function(uri, requestBody)
    {
        invoice = getGreenInvoiceDataFromRequest(requestBody);
        expect(invoice.params.doc_type).to.equal(320);
        expect(invoice.params.client.name).to.equal(newPayment.transaction.invoice.recipient);
        expect(invoice.params.income[0].price).to.equal(newPayment.sum);
        expect(invoice.params.income[0].description).to.equal(newPayment.transaction.invoice.item);
        expect(invoice.params.payment[0].type).to.equal(1);
        expect(invoice.params.payment[0].amount).to.equal(newPayment.sum);

        // return a response indicating success
        return {'error_code': 0, data: {ticket_id: '8cdd2b30-417d-d994-a924-7ea690d0b9a3'}};
    });

MEAN (Angular, Node) for Django programmers (part 1)

A few things I would have found helpful when making the switch from Django/Twisted to MEAN (Mongo, Express, Angular, Node).

How should I get started?

For me the process was:

  1. Write a simple RESTful service with Node/Express/Mongo
  2. Write a standalone, server-less Angular application
  3. Combine the two together

To get start with (1), I recommend this scotch.io tutorial. There's nothing conceptually new here besides perhaps Mongo being a document store, but this won't come into play; mongoose wraps that up like an ORM.

You should then follow the official Angular tutorial to get a grasp on Angular. I skipped the whole testing portion and left it for later (this is 50% of the tutorial). I wanted to get something working and get a feel for Angular before I invested effort familiarizing myself with automated testing tools. The important thing to realize here is that Angular lets you build self contained single page applications that run in your browser. While it is often very useful to consume remote service APIs, this isn't a must. You can get quite a lot of functionality without accessing a single remote server.

Once you have some feel for the above, think about something useful you'd like to build (more complex than a todo app). Define a RESTful service (that will most likely just handle authentication and object storage) and implement an Angular application that consumes it (or alternatively, follow this scotch.io tuturial. Have Node serve both the RESTful service endpoints and the Angular JS files. A user will then navigate to your server and pull the Angular application and its dependencies (possibly only after authenticating). The application, in turn, will communicate with the RESTful service and populate itself with data.

Should I use a framework like mean.io or roll my own?

I have re-invented many wheels in my day but when I dive into something new, I like to adopt widely used infrastructures - the automatic choice, the road most travelled. At first I thought mean.io is the go to framework for MEAN, but I found this not to be the case. There were two reasons why I decided to build from the ground up rather than carve my application out of mean.io:

  1. Mean.io experienced a bump in the road when their main guy jumped ship and forked out to Mean.js. The company maintaining Mean.io then refactored it heavily (quite a lot of people critisized mean.io for its code quality pre-refactoring). All of this it didn't add up to a healthy place to start
  2. The basic boilerplate mean.io project is quite large and (as its creator mentioned) seems more like a hackathon seed than a clean base with which to create focused projects, especially when learning the ropes

This may mean a bit more work and some boilerplate code but at this early stage I think this is a good thing. I prefer wiring things up to learn how they work before using infrastructures that hide all the details.

Which seed project should I use for MEAN?

If, like me, you decide not to go the mean.io route you will quickly find that it's up to you to decide how your code directory structure will look like. Django decides this for you (for better or worse) but with MEAN you are left to decide which seed project directory structure your code will follow. You may be tempted to follow Google's angular-seed as it is widely known and used in many Angular tutorials. This is a great choice to start, but is simply awful for anything more than a test project.

At first I tried looking at mean.io for inspiration but quickly found it too bloated and fragmented even for that (again, assuming we're trying to learn from the ground up rather than just making it work).

Two things eventually influenced the directory structure I used in my first MEAN project:

  1. This article by Cliff Meyers made a lot of sense to me regarding how to structure the Angular code. I felt he made a good point that this structure describes what the application does, and it also felt like something that could grow horizontally (more files) rather than vertically (larger files)
  2. Quite a few seeds I looked at (which were well received on Github) separated the client/ from the server/ and that felt like a natural thing to do

I have no doubt that the structure I chose will change (and perhaps even evolve into something similar to mean.io or mean.js) but for a small personal-yet-deployed project this held up pretty well.

What is the best development environment?

This is easy as there are no worthy alternatives - use Webstorm. It's far from perfect but it's pretty damn good. The editor may not be as good as Sublime Text but the fact that you can easily source level debug both when running tests and running a development server is a must. Do not be tempted to debug your code with the logger. The minute you write actual code that does something you need source level debugging.

More insights in part 2, here.