Using robocopy to copy folder without node_modules subfolder(s) and using WinMerge and TreeSize to check

Note: This post is a details-post for the post: Learning web app development through free online tutorials – Organized Notes and Log, https://raviswdev.blogspot.com/2024/03/learning-web-app-development-through.html .

React and Node program directories have lots of files due to node_modules directory. For backup purposes, copying main source code of these directories is a challenge as node_modules should ideally be skipped. Individual project directory copy can be done manually by excluding node_modules - that is no problem. The problem crops up for backing up many such projects (in my case, typically as part of a backup done after 2 to 3 months of all required data to USB backup drive) as manual exclusion will be too time-consuming. 

Windows command line utilities like robocopy can be used but I had a concern that if I made a mistake in the command specification, the copy may not be exactly what I want. Ideally I should be able to confirm that the copied folder and its subfolders are as expected, using some top-level control data like number of files and folders and size and perhaps also visually be able to quickly see that only node_modules folder has been excluded from the output. 

Note that node_modules folder can be deleted before copy and then re-created using npm install. But npm install downloads all the modules again which takes some time. There is also a minor issue that for some of the tutorial projects I have been using, npm install followed by npm start can show some errors like "digital envelope routines::unsupported". Usually the fix for these errors is straightforward but ... The point is that if I leave the node_modules folder as is for some projects that I think I may be revisiting/reusing in a short period of time (so long as I have enough space on my SSD drive on PC), I can run the associated program quickly when I want by npm start. So for some projects (certainly not all), I prefer not to delete their node_modules folder.

I recently got a solution that seems to work for me and thought of putting up a post on it as it may be of some help to others too. The contents below are based on a note I prepared for my future needs. Readers would need to modify the paths suitably to point to their projects folder.

Procedure to copy Projects folder without node_modules folder(s) to backup drive including checking that the backup copy folder is as expected

1) Robocopy command to create a copy of D:\Users\Ravi-user\Projects data but without node_modules in directory named Projects-WO-node_modules is as follows (can be run in Command Prompt window):

robocopy D:\Users\Ravi-user\Projects D:\Users\Ravi-user\Projects-WO-node_modules /E /XD node_modules

Note the statistics data provided on command completion on console.

2) How to use WinMerge folder compare on Projects and Projects-WO-node_modules and visually confirm that node_modules folders missing from Projects-WO-node_modules are the only differences between the two folders:

2a) Run Winmerge and set Edit->Options->Compare->Folder option "Include unique subfolders contents" as unchecked (it is checked by default). This seems to skip going through node_modules of Projects folder and so the comparison is finished much faster. Compare the two folders (User guide page). Only subfolders that should be reported as different should be those where Projects folder entry (subfolder) has node_modules and Projects-WO-node_modules does not have it. This can be confirmed by drilling down the entries saying folders are different till Comparison result: 'Left only {folder-name}' with folder-name being the folder containing that node_modules folder, is seen. That is the cause of its parent folder(s) being reported as different. Other than node_modules, no folder or file should be having a Comparison result of 'Left only ...'. 

3) How to use TreeSize Free software's stats. data (number of files, folders and their size) to check whether Robocopy command worked as expected:

3a) Open D:\Users\Ravi-user\Projects in TreeSize and use filter of: "Include the following files", "And", "Any Parent Folder Name", "does not equal", "node_modules".  This will give Projects folder size and number of files without node_modules folders and their nested contents. The number of files copied figure in end of run statistics of robocopy should be equal to the number of files shown by TreeSize (with above filter). The Bytes copied figure of robocopy should be very close to or be equal to size figure of TreeSize. The number of Dirs copied figure of robocopy may be 1 more than the folders figure of TreeSize (this could be due to Projects-WO-node_modules folder being counted by robocopy but not by TreeSize.

3b) Open D:\Users\Ravi-user\Projects-WO-node_modules in another TreeSize invocation/window and do not use any filter (clear filter if any). The Size, Files and Folders of both the above TreeSize windows should be the same.

4) After WinMerge and TreeSize checks confirm that D:\Users\Ravi-user\Projects-WO-node_modules folder is as expected, the folder can be simply copied over to Backup drive using Windows Explorer copy. Optionally, after that copy is done, TreeSize can be run on backup drive folder and its number of files, folders and size should be same as for D:\Users\Ravi-user\Projects-WO-node_modules folder.

Notes

i) When I used a variation of robocopy step 1 above to directly copy files to backup drive (USB drive), I faced a problem. After the copy got over successfully and I moved the copied directories to another new folder I created on the backup drive, sometime later on same day, I got a data corruption message for the backup drive. The folder I had created on backup drive into which I had copied the robocopy output folder(s) became inaccessible and later disappeared! I had to fix the data corruption warning by 'repair'ing the disk. The folder which had disappeared did not appear again. Essentially, the robocopy copied data all got lost! I don't know what the issue was - could be USB disk issue. But I wondered whether using robocopy to copy directly to USB drive was the issue. So I modified the procedure to do the robocopy step on SSD (output folder) in PC and then copy the output folder from SSD to USB drive using standard Windows Explorer copy.

========================================================