Notes on quick 1st round of remaining part of John Smilga Node Express tutorial (after Tasks project)

Last updated on 9 Oct. 2024
2 & 3 Oct. 2024:
Continuing from after Tasks project in John Smilga Express tutorial...Node.js Projects,
https://youtu.be/rltfdjcXjmk?t=11235 (from around 3 hrs, 7 mins in the video).

The next project taken up is the Store project.


Most of the source code of the Store project is straight-forward and somewhat similar to Tasks project. However, controllers\products.js has fair deal of complexity as it uses the various query features of Mongoose find() with Query String params being the mechanism for specifying the query to the get route ('routed' to getAllProducts() method in controllers\products.js file).

The code for string data query is simple. Numeric data query code is complex which is covered below in some detail.

Chaining the find method conditionally with sort and select methods, and unconditionally with skip and limit methods is new in the tutorial. The await keyword is not used with find but used with a 'result' object that holds return value of the earlier chained methods.

https://mongoosejs.com/docs/api/model.html#Model.find() examples do not cover such chaining. It instead shows three parameters for find() - filter (having the search conditions part), projection (having the select part - https://mongoosejs.com/docs/api/query.html#Query.prototype.select() ) and options (having the limit, skip and sort (and more options) - https://mongoosejs.com/docs/api/query.html#Query.prototype.setOptions() ). IIRC, the tutorial video does cover such find() usage too but then says that dynamic addition of sort and select could be done using the approach it takes (and, IIRC, cannot be done using only parameters without chaining).

How find() Works in Mongoose, https://thecodebarbarian.com/how-find-works-in-mongoose, Feb. 2019 - covers find() with parameters and with chaining. Article is very helpful to get better understanding of these topics.
...

In controllers\products.js (in final version) of Store project:
      if (options.includes(field)) {
        queryObject[field] = { [operator]: Number(value) };
      }
----
As I understand it, if only operator is used then the property name would be operator. We need property name to be the value of the operator variable and so use [operator]. See references below:

"The object initializer syntax also supports computed property names. That allows you to put an expression in square brackets [], that will be computed and used as the property name."

What do square brackets around a property name in an object literal mean?, https://stackoverflow.com/questions/34831262/what-do-square-brackets-around-a-property-name-in-an-object-literal-mean has an interesting answer about using "key"*5 as property name. Without square brackets it would result in syntax error, it says. ["key"*5] works!

===============
In controllers\products.js (in final version) of Store project:
  if (numericFilters) {
    const operatorMap = {
      '>': '$gt',
      '>=': '$gte',
      '=': '$eq',
      '<': '$lt',
      '<=': '$lte',
    };
    const regEx = /\b(<|>|>=|=|<|<=)\b/g;
    let filters = numericFilters.replace(
      regEx,
      (match) => `-${operatorMap[match]}-`
    );
    const options = ['price', 'rating'];
    filters = filters.split(',').forEach((item) => {
      const [field, operator, value] = item.split('-');
      if (options.includes(field)) {
        queryObject[field] = { [operator]: Number(value) };
      }
    });
  }
----------
https://hn.algolia.com/api has examples of conditional search which style seems to have been used by Smilga tutorial (with attribution).
"Stories between timestamp X and timestamp Y (in second)
...
Decoding regex: /\b(<|>|>=|=|<|<=)\b/g
\b word boundary  Ref: https://www.regular-expressions.info/wordboundaries.html - 'Simply put: \b allows you to perform a “whole words only” search using a regular expression in the form of \bword\b. A “word character” is a character that can be used to form words. All characters that are not “word characters” are “non-word characters”.'
Ref: https://www.regular-expressions.info/shorthand.html "\w stands for “word character”. It always matches the ASCII characters [A-Za-z0-9_]. Notice the inclusion of the underscore and digits."
My understanding is that < > =  are NOT word characters [confirmed by https://regex101.com/ test]

| is used for alternation (like OR) Ref: https://www.regular-expressions.info/alternation.html
( and ) are used for grouping  Ref: https://www.regular-expressions.info/brackets.html
g at end stands for global    From https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/global : "The g flag indicates that the regular expression should be tested against all possible matches in a string."
...
So /\b(<|>|>=|=|<|<=)\b/g seems to mean:
find/get global matches for < or > or >= or = or < or <= which has word boundaries to its left and right.
...
Tested /\b(<|>|>=|=|<|<=)\b/g  in https://regex101.com/
price<2,rating>4.5 - has 2 matches < and >
price<<2,rating>4.5 - has only one match >
price<<2,rating>=4.5 - has only one match >=
-----------

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace : "The replace() method of String values returns a new string with one, some, or all matches of a pattern replaced by a replacement. The pattern can be a string or a RegExp, and the replacement can be a string or a function called for each match. If pattern is a string, only the first occurrence will be replaced. The original string is left unchanged." ... "If (replacement is) a function, it will be invoked for every match and its return value is used as the replacement text."
----

// find all documents named john and at least 18
await MyModel.find({ name: 'john', age: { $gte: 18 } }).exec();
---

In above controllers\products.js code, my understanding of it is that if numericFilters is "price>20,rating<=4.5" then it will be processed as follows (see // (comment) lines in code below):
  if (numericFilters) {
    const operatorMap = {
      '>': '$gt',
      '>=': '$gte',
      '=': '$eq',
      '<': '$lt',
      '<=': '$lte',
    };
    const regEx = /\b(<|>|>=|=|<|<=)\b/g;
    let filters = numericFilters.replace(
      regEx,
      (match) => `-${operatorMap[match]}-`
    );
// Above code will have two matches and filters will become "price-$gt-20,rating-$lte-4.5"
    const options = ['price', 'rating'];
    filters = filters.split(',').forEach((item) => {
      const [field, operator, value] = item.split('-');
// field: price, operator: $gt, value: 20
// field: rating, operator: $lte, value: 4.5
      if (options.includes(field)) {
        queryObject[field] = { [operator]: Number(value) };
// Following properties and associated values will be added to queryObject:
// price: {$gt : 20}
// rating: {$lte : 4.5}
// The above is what is required for mongoose find().
      }
    });
  }
---------
=============================

4 Oct. 2024:

https://mongoosejs.com/docs/queries.html which is part of the Mongoose docs guide covers chaining and parameters equivalent example for find().

About JWT-Basics project in Express tutorial: https://youtu.be/rltfdjcXjmk?t=18330 (from around 5 hrs, 5 mins. in the video)
The code is quite straight-forward. Some notable points:

jwt.io - useful site.

The tutorial does not send back the jwt token as a cookie. Instead it sends it back as part of the json response for the login route. Also, the dashboard route expects the token to be specified as part of the Authorization header (Bearer token).

https://www.loginradius.com/blog/engineering/guest-post/nodejs-authentication-guide/ sends the token to the client as a cookie on successful login or registration. Protected routes then check for token in the cookies sent with the request.

Don't know which is the more appropriate approach for JWT. Here are some posts/articles on it got from Google search, but in my quick read, I did not get a clear answer. There is also some confusion about JWT being stored in cookie and used for authentication and using a cookie without JWT for authentication.
  1. Should JWT token be stored in a cookie, header or body, https://security.stackexchange.com/questions/130548/should-jwt-token-be-stored-in-a-cookie-header-or-body - Looks like this post has the best coverage among these posts/articles for the question I have raised.
  2. Section "Comparison of JWT and Cookies storage" in https://strapi.io/blog/introduction-to-jwt-and-cookie-storage
  3. JWT vs cookies for token-based authentication, https://stackoverflow.com/questions/37582444/jwt-vs-cookies-for-token-based-authentication

In middleware\auth.js :
const { UnauthenticatedError } = require('../errors')
----
errors is a folder which has an index.js file whose contents are:
const CustomAPIError = require('./custom-error')
const BadRequestError = require('./bad-request')
const UnauthenticatedError = require('./unauthenticated')

module.exports = {
  CustomAPIError,
  BadRequestError,
  UnauthenticatedError,
}
---
custom-error.js, bad-request.js and unauthenticated.js files are present in errors folder.

Node docs explains how it works. First https://nodejs.org/api/packages.html#modules-loaders states that CommonJS module loader "supports folders as modules" and links to https://nodejs.org/api/modules.html#folders-as-modules which states the way folders as modules works with one step being node looking for an index.js file in the folder and loading it if found.
======================================
 
Oct 5. 2024

In the JWT-Basics project, the frontend stores the token in Local Storage, and sends the token as an authorization header for the dashboard route get request. Related code from public/browser-app.js:

btnDOM.addEventListener('click', async () => {
  const token = localStorage.getItem('token')
  try {
    const { data } = await axios.get('/api/v1/dashboard', {
      headers: {
        Authorization: `Bearer ${token}`,
      },
    })
    resultDOM.innerHTML = `<h5>${data.msg}</h5><p>${data.secret}</p>`
...
-------

https://johnsmilga.com/  , https://johnsmilga.com/projects - the tut video says that source code links are provided in these pages but that does not seem to be the case as of now.

We can checkout the public repos of John Smilga on Github -  https://github.com/john-smilga - but that's not as straightforward as source code link in the projects pages.

===========================
6 & 7 Oct. 2024
Jobs API project (06-jobs-api)

In the initial part, it covers Registration (user, email, (hashed) password stored in MongoDB collection) and then Login. It also uses JWT. So this project code could be a very useful reference to implement similar Registration and Login functionality in some other project.
...
In models\User.js, mongoose middleware - pre save is used for hashing password before writing to User collection.
mongoose instance method (createJWT) is used to return a JWT token. This method is invoked by register method in controllers\auth.js

Note that arrow functions are not used in above mongoose middleware or instance method as the 'this' keyword in arrow functions has somewhat different behaviour. We need to access current document in the function which can be done using this keyword in plain 'function. So plain 'function' is used. [I am choosing not to get into the details of arrow function this keyword as I don't think I need to know those details now.]

Broken link: allkeysgenerator.com to generate JWT secret. Choose Encryption key, 256-bit,
Checked out alternatives via Google search: https://acte.ltd/utils/randomkeygen seems to provide "Encryption key 256" which may meet JWT secret needs.

=========

A little before 8 hrs, 29 mins into the tutorial video, the tests feature of Postman is used for login and register routes to code a few lines of JavaScript to extract token after the API is 'sent' and save the token to a Postman global variable. This global variable is then used in Authorization tab of other routes like Get and Create as a Bearer token. This approach avoids having to copy-paste the token value into header of routes like Get and Create. Smilga refers to this as "dynamically" setting the token in Postman.
---------------

Nested destructuring ... Associated code from controllers\jobs.js
  const {
    user: { userId },
    params: { id: jobId },
  } = req

  const job = await Job.findOne({
    _id: jobId,
    createdBy: userId,
  })
--------------
jobId is used as an alias for id destructured from params.
==============

Security packages:
helmet
cors
xss-clean
express-rate-limit

In app.js
app.set('trust proxy', 1);
----
is required for Heroku which uses/is a reverse proxy. For more, see express-rate-limit docs.

==========
Swagger UI is used to generate documentation but with a procedure involving Postman.
Postman routes for Jobs collection are used to create an export JSON file.
This JSON file is imported into APIMatic.io which generates API (description/file) ... Some config settings are done including specifying URL of live Heroku jobs api server, modifying authentication settings to reflect that register and login are not protected routes.
Then apimatic.io API is exported as OPENAPI v3.0 (YAML).

That's input to Swagger UI editor. Some editing is needed to get what we want finally. We get API documentation page along with testing of the APIs facility.

Next step is to add this documentation to the API project itself. ... yamljs and swagger-ui-express packages are needed. The swagger edited file is copy-pasted into the project as swagger.yaml

In app.js:
const swaggerUI = require('swagger-ui-express');
const YAML = require('yamljs');
const swaggerDocument = YAML.load('./swagger.yaml');

...

app.get('/', (req, res) => {
  res.send('<h1>Jobs API</h1><a href="/api-docs">Documentation</a>');
});
app.use('/api-docs', swaggerUI.serve, swaggerUI.setup(swaggerDocument));

----------
That finishes the Node Express ... Node.js Projects tutorial video. Next I plan to go through all of the source code of Jobs API project (06-jobs-api).
===============

8 Oct. 2024
https://www.npmjs.com/package/helmet - "Help secure Express apps by setting HTTP response headers."
https://www.npmjs.com/package/cors - "CORS is a node.js package for providing a Connect/Express middleware that can be used to enable CORS with various options."
xss-clean used in tutorial project has been deprecated: https://www.npmjs.com/package/xss-clean - "This library has been deprecated. The implementation is quite simple, and I would suggest you copy the source code directly into your application using the xss-filters dependency, or look for alternative libraries with more features and attention. Thanks for your support." ... "Node.js Connect middleware to sanitize user input coming from POST body, GET queries, and url params. Works with Express, Restify, or any other Connect app."
https://www.npmjs.com/package/express-rate-limit - "Basic rate-limiting middleware for Express. Use to limit repeated requests to public APIs and/or endpoints such as password reset. Plays nice with express-slow-down and ratelimit-header-parser."

https://www.npmjs.com/package/swagger-ui-express - "This module allows you to serve auto-generated swagger-ui generated API docs from express, based on a swagger.json file. The result is living documentation for your API hosted from your API server via a route."
https://www.npmjs.com/package/yamljs - "Standalone JavaScript YAML 1.2 Parser & Encoder. Works under node.js and all major browsers. Also brings command line YAML/JSON conversion tools."
The "Load swagger from yaml file" section of https://www.npmjs.com/package/swagger-ui-express is relevant for tutorial code.

Express behind proxies, https://expressjs.com/en/guide/behind-proxies.html : "When running an Express app behind a reverse proxy, some of the Express APIs may return different values than expected. In order to adjust for this, the trust proxy application setting may be used to expose information provided by the reverse proxy in the Express APIs. The most common issue is express APIs that expose the client’s IP address may instead show an internal IP address of the reverse proxy."


https://www.npmjs.com/package/jsonwebtoken shows asynchronous and synchronous sign function usage. Tutorial code uses sync sign function.

---------
middleware\error-handler.js has lot of additional code for error handling.
I searched for Mongoose error handling to see if there is a good reference or guide page on Mongoose that covers error handling. I could not get one page/location giving that. But I have been able to go through the Mongoose docs rather painstakingly and get some understanding of Mongoose error handling.

https://mongoosejs.com/docs/api/error.html#Error() states "MongooseError constructor. MongooseError is the base class for all Mongoose-specific errors." It also states that it inherits from JavaScript Error class, https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error.

It is this JS Error class that provides name property. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/name - 'The name data property of Error.prototype is shared by all Error instances. It represents the name for the type of error. For Error.prototype.name, the initial value is "Error".'

Mongoose Error.prototype.name, https://mongoosejs.com/docs/api/error.html#Error.prototype.name gives a list of names that Mongoose uses for its errors. Two of these error names are relevant for tutorial:
CastError: Mongoose could not convert a value to the type defined in the schema path. May be in a ValidationError class' errors property.
ValidationError: error returned from validate() or validateSync(). Contains zero or more ValidatorError instances in .errors property.

Validation, https://mongoosejs.com/docs/validation.html has some example code which is useful to understand the errors thrown by Mongoose. The examples use validateSync method which is covered here: https://mongoosejs.com/docs/api/document.html#Document.prototype.validateSync() - "Executes registered validation rules (skipping asynchronous validators) for this document." Also the validation.html page states, "You can manually run validation using doc.validate() or doc.validateSync()".

https://mongoosejs.com/docs/validation.html#validation-errors - "Errors returned after failed validation contain an errors object whose values are ValidatorError objects. Each ValidatorError has kind, path, value, and message properties."

But there is Mongoose ValidationError and ValidatorError! That tripped me up initially. Now I think I am getting a hang of it.

Error.ValidationError, https://mongoosejs.com/docs/api/error.html#Error.ValidationError - "An instance of this error class will be returned when validation failed. The errors property contains an object whose keys are the paths that failed and whose values are instances of CastError or ValidationError."

Error.ValidatorError, https://mongoosejs.com/docs/api/error.html#Error.ValidatorError - "A ValidationError has a hash of errors that contain individual ValidatorError instances." It has the following code example which explains well the difference between ValidationError and ValidatorError objects:
const schema = Schema({ name: { type: String, required: true } });
const Model = mongoose.model('Test', schema);
const doc = new Model({});

// Top-level error is a ValidationError, **not** a ValidatorError
const err = doc.validateSync();
err instanceof mongoose.Error.ValidationError; // true

// A ValidationError `err` has 0 or more ValidatorErrors keyed by the
// path in the `err.errors` property.
err.errors['name'] instanceof mongoose.Error.ValidatorError;

err.errors['name'].kind; // 'required'
err.errors['name'].path; // 'name'
err.errors['name'].value; // undefined
----

path is the property/field in the schema.

============
In middleware\error-handler.js we have:
const errorHandlerMiddleware = (err, req, res, next) => {
...
  if (err.name === 'ValidationError') {
    customError.msg = Object.values(err.errors)
      .map((item) => item.message)
      .join(',')
    customError.statusCode = 400
  }
  if (err.code && err.code === 11000) {
    customError.msg = `Duplicate value entered for ${Object.keys(
      err.keyValue
    )} field, please choose another value`
    customError.statusCode = 400
  }
  if (err.name === 'CastError') {
    customError.msg = `No item found with id : ${err.value}`
    customError.statusCode = 404
  }
------

If it is ValidationError, the err.errors object, as per my understanding, is a hash of ValidatorError objects, and, as per my understanding, this hash is a hash table kind of data structure.
The Object.values() static method returns an array of a given object's own enumerable string-keyed property values.
...
If you need the property keys, use Object.keys() instead. If you need both the property keys and values, use Object.entries() instead.
----

Object.values(err.errors) should return an array of the values of keys in the hash of ValidatorError objects err.errors. 

From above - "Each ValidatorError has kind, path, value, and message properties." So each such value is an object with keys of kind, path, value, and message and associated values for these keys.
map((item) => item.message) picks up the message key's value from each such value object, and so we get an array of message strings which is then joined with a comma.

The Duplicate value error handling code uses err.code. I could not locate a manual page of Mongoose or JavaScript (MDN) which shows code as a property of the Mongoose or JavaScript error object. Note that code seems to be an optional property as the source-code first checks whether it is defined - if (err.code && err.code === 11000)

Mongoose MongoError : 11000, https://stackoverflow.com/questions/68174025/mongoose-mongoerror-11000 gives an example of the error with the error object data, which I have copy-pasted below:
Error creating new record : {
  "driver": true,
  "name": "MongoError",
  "index": 0,
  "code": 11000,
  "keyPattern": {
    "RoutineParts.userId": 1
  },
  "keyValue": {
    "RoutineParts.userId": null
  }
}
----

The "MongoError" name is not listed in Mongoose Error prototype names: https://mongoosejs.com/docs/api/error.html#Error.prototype.name

Perhaps it is an error generated by MongoDB which Mongoose simply allows to go up to the relevant catch code. I searched for it in MongoDB docs but did not get a simple page explaining it. I think I should not invest further time on trying to get MongoDB docs page for it.

In the tutorial code:
customError.msg = `Duplicate value entered for ${Object.keys(err.keyValue)} field, please choose another value`
----
as I understand it, going by the "MongoError" related error data given above, Object.keys(err.keyValue) should return an array of 1 element which is a string having name of the field. [Object.keys ref: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/keys]

If it can only be one field then perhaps the code could have been:
customError.msg = `Duplicate value entered for ${(Object.keys(err.keyValue)[0])} field, please choose another value`
----
But I am not sure about this.
...
About the 'castError' case in tutorial code, repeated below:
  if (err.name === 'CastError') {
    customError.msg = `No item found with id : ${err.value}`
    customError.statusCode = 404
  }
---
err.name comes from JavaScript error object property which is set appropriately by Mongoose.
But where does the err.value property come from?

https://mongoosejs.com/docs/api/error.html#Error.CastError states, "An instance of this error class will be returned when mongoose failed to cast a value." But it does not say anything about contents of this class-object. 

https://mongoosejs.com/docs/validation.html#cast-errors has example code which is quite different from tutorial code:
const vehicleSchema = new mongoose.Schema({
  numWheels: { type: Number, max: 18 }
});
const Vehicle = db.model('Vehicle', vehicleSchema);

const doc = new Vehicle({ numWheels: 'not a number' });
const err = doc.validateSync();

err.errors['numWheels'].name; // 'CastError'
// 'Cast to Number failed for value "not a number" at path "numWheels"'
err.errors['numWheels'].message;
----
Tutorial code uses err.name to check for 'CastError' whereas above doc. uses err.errors[path or field name].name.
Hmm.
Tutorial uses Mongoose 5.13.2 whereas I am looking up current (8.7.0) Mongoose docs. But "Cast Errors" section of Mongoose 5.13.21, https://mongoosejs.com/docs/5.x/docs/validation.html has similar/same code as above!

How to handle for Cast Errors in MongoDB/mongoose, https://stackoverflow.com/questions/64693832/how-to-handle-for-cast-errors-in-mongodb-mongoose , Nov. 2020 has similar code to that of the tutorial. It uses err.name, err.path and err.value!

I don't want to invest further time on this, as of now.
======================

In controllers\jobs.js:
  const job = await Job.findOne({
    _id: jobId,
    createdBy: userId,
  })
----

That this is mapped to AND is confirmed by MongoDB docs: Specify AND Conditions, https://www.mongodb.com/docs/manual/tutorial/query-documents/#specify-and-conditions 

=========
While job model has:
    status: {
      type: String,
      enum: ['interview', 'declined', 'pending'],
      default: 'pending',
    },
----
the updateJob controller function does not look for a request body status property and so the API does not seem to provide a way to change the status from initial default value.

====
In controllers\jobs.js in deleteJob() function:
  const job = await Job.findByIdAndRemove({
    _id: jobId,
    createdBy: userId,
  })
----

findByIdAndRemove is present in Mongoose 5.x, https://mongoosejs.com/docs/5.x/docs/api/model.html#model_Model.findByIdAndRemove  but does not seem to be available in Mongoose 8.7 (current version), https://mongoosejs.com/docs/api/model.html .

findByIdAndRemove 5.x docs states:
Parameters
id «Object|Number|String» value of _id to query by
[options] «Object» optional see Query.prototype.setOptions()
...
----
So id can be passed as an object! I think to understand this well, the underlying MongoDB function which seems to be findAndModify will need to be studied and that may be deprecated. Not worth investing time in reading up this.

My guess is that the above code has a bug as the createdBy property in passed object may be ignored by findByIdAndRemove (and so even if the user has not created that particular job, it will be deleted).

In current version perhaps the following may be appropriate: findOneAndDelete() - https://mongoosejs.com/docs/api/model.html#Model.findOneAndDelete()
=============

This project does not have any front-end code.
I think that finishes (on 8th Oct. 2024) this quick 1st round of these parts of the tutorial video (after Tasks project to end of video which is end of Jobs project).
===================
The full tutorial source code has additional projects: 07-file-upload, 08-send-email, 09-stripe-payment, 10-e-commerce-api and 11-auth-workflow. ... https://github.com/john-smilga/node-express-course . But the video for these parts seems to be behind a paywall : https://www.udemy.com/course/nodejs-tutorial-and-projects-course/?couponCode=NVDIN35 . Google search brought up this related post which confirms my previous presumption: Need help whether to buy John Smilga Nodejs course on Udemy ?, https://www.reddit.com/r/node/comments/172tw9h/need_help_whether_to_buy_john_smilga_nodejs/, Oct. 2023 (see baton912's update message,  https://www.reddit.com/r/node/comments/172tw9h/comment/k3zlvau/ ).

If I want and have the time, I could go through the source code of these projects to get an idea about them.

=================================
9 Oct. 2024, Some additional info.

Auth.js for Express is currently experimental - https://authjs.dev/reference/express

Authentication strategies available in Express, https://www.geeksforgeeks.org/authentication-strategies-available-in-express/ , last updated Dec. 2023 : Has quick coverage of:
1) Stateless Authentication
  • Basic Authentication using express-basic-auth 
  • Token-Based Authentication (JWT) using jsonwebtoken
  • OAuth Authentication using passport and passport-google-oauth20
2) Stateful Authentication - The examples given for this type of authentication were not clear to me. The examples cover Passport.js middleware and custom middleware. The intro mentions that cookies are used to identify the user but the examples code does not involve any cookies. Perhaps passport.js middleware handles the cookies part but even the custom middleware example does not use any cookies.

Comments

Post a Comment