Revisioning asset files

This chapter is about the ultimate level of build complexity, ASSET revisioning. 99,9 % of all websites display some kind of graphics or other kinds of program asset data that is not source code of any kind like CSS, Javascript or HTML. Nevertheless, this data can change, too. Every single picture or asset for a website is seperately downloaded by the browser. To prevent unnecessary downloads, also website assets are stored in the browser’s cache. Therefore, we need to trigger the cache buster in case our assets change.

The difference to what we accomplished in the previous chapter is that all assets are referenced in source code whereas the references to CSS, HTML and Javascript files were all stored in index.html. In index.html, we conveniently could place our file reference within special comments where they could easily be found. Assets, on the other hand, are referenced somewhere is our code files. References to assets like images or fonts can be found in CSS and Javascript files. If the name of an asset changes (e.g., because of revisioning), someone must scan through all CSS and Javascript files and update those references with the new filenames.

This is now really getting complex, so I will try to make it as simple as possible.  Imagine you have a bunch of files which are renamed because of revisioning. You might think that a plugin could take the first renamed file and just goes through all source files and scans for the original filename in order to replace it with the new one. Then the plugin would do the same for the next file that is renamed, and so on. If you have hundreds of asset file, this process could drag on. Wouldn’t it be better to rename the affected files first and then scan through the source files just once? For this to work, the plugin must maintain a file which holds the original filenames mapped to the corresponding renamed ones. In our case, the gulp-rev plugin can optionally create such a file. You can freely specify the name of this file, but the developers of gulp-rev started naming it to some kind of “manifest” file, but don’t mistaken it with the HTML5 application cache manifest file (its relationship to the gulp-rev manifest file is explained in the next chapter). What we need to know right now is that we name this file rev-manifest.json and that this file is created and maintained by the gulp-rev plugin.

Please note that gulp-rev only creates this mapping file. It does not do any source file scanning for references to files listed in it. This is done by another plugin. FOr now, let’s see an example. Assume that somewhere in our sources we reference some icon files. The files are stored in our project’s source folder somewhere (preferably in a dedicated ‘assets’ folder:

"pill_green_sm.png"
"pill_grey_sm.png"
"pill_red_sm.png"
"pill_yellow_sm.png"

These icon files may change someday in the future, not likely, but possible. We want to revision these files so that the new version will definetely pierce through the browser’s cache. After being processed by the gulp-rev plugin, the new names might be

"pill_green_sm-4b8546f4db.png"
"pill_grey_sm-6c162f4b40.png"
"pill_red_sm-27795e8821.png"
"pill_yellow_sm-93662c81c0.png"

Keep in mind that the original files must be preserved (e.g., for testing), just like all other source code files of our web app. Like all other build-processed files, we need to make sure that they are going into our project’s dist folder. Since we already included gulp-rev in our package.json and Gulpfile.js,  we can jump right to our next Gulp task, which we’ll  call “assets”. See the call to rev.manifest(). That’s where all assets from the rev() step are listed in a manifest file called rev-manifest.json.

gulp.task('assets', function () {
    return gulp.src(['src/assets/**/*'])
        .pipe(rev())
        .pipe(gulp.dest('dist/assets'))
        .pipe(rev.manifest('/MyStunningWebProject/dist/rev-manifest.json', {
            merge: true,
            base: '/MyStunningWebProject/dist'
        })) // merge with the existing manifest (if one exists)
        .pipe(gulp.dest('dist/'));
});

You can run this task by entering into the command line:

c:> gulp assets

After this task has completed, we should see rev-manifest.json in our dist folder. If we had the icon files as our only assets, this is what’s inside rev-manifest.json:

{
  "pill_green_sm.png": "pill_green_sm-4b8546f4db.png",
  "pill_grey_sm.png": "pill_grey_sm-6c162f4b40.png",
  "pill_red_sm.png": "pill_red_sm-27795e8821.png",
  "pill_yellow_sm.png": "pill_yellow_sm-93662c81c0.png"
}

Don’t be disappointed, but we are not there yet. What happened so far is that we got the asset files renamed and stored into dist, so nothing new. What is new is that we have a file called rev-manifest.json which contains the mapping between original filename and new ones. For simplicity sake, we didn’t create multiple manifest files for different assets. This is possible and makes sense in some cases. We will continue with only a single manifest file. With rev-manifest.json we are now ready to scan through our source files and replace the original filenames with the ones mapped in rev-manifest.json. To accomplish that, we need another task which reads the manifest files in our project and does the required modifications. The plugin for doing that is called gulp-rev-collector.

c:> npm install gulp-rev-collector --save-dev

gulp.task('collect', ['assets'], function () {
    return gulp.src(['dist/rev-manifest.json', 'dist/**/*.js', 'dist/**/*.css', 'dist/**/*.html'])
        .pipe(collect())
        .pipe(gulp.dest('dist/'));
});

The collect task must only run after the assets task has completed. So in line 1 we specified the corresponding dependency with ['assets'].

How to properly sequence all our tasks in Gulpfile.js?

The current usemin task creates the files which are targeted by the collect task. Therefore, the new tasks assets and collect must run after usemin. Because collect needs the manifest file created during the assets task, it must be run after assets has finished. If we want to fully automate our build process, we need to properly sequence our existing tasks.

1. clean
2. assets
3. usemin
4. collect << our entry point to the build pipeline

This is what the complete Gulpfile.js looks so far:

var gulp = require("gulp");
var clean = require("gulp-clean");
var uglify = require("gulp-uglify");
var usemin = require('gulp-usemin');
var minifyHtml = require('gulp-minify-html');
var minifyCss = require('gulp-minify-css');
var rev = require('gulp-rev');
var collect = require('gulp-rev-collector');

gulp.task('collect', ['usemin'], function () {
    return gulp.src(['dist/rev-manifest.json', 'dist/**/*.js', 'dist/**/*.css', 'dist/**/*.html'])
        .pipe(collect())
        .pipe(gulp.dest('dist/'));
});

gulp.task('usemin', ['assets'], function () {
    return gulp.src('src/index.html')
        .pipe(usemin({
            css: [minifyCss(), rev()],
            html: [minifyHtml({empty: true})],
            js: [uglify(), rev() ] }))
        .pipe(gulp.dest('dist/'));
});

gulp.task('assets', ['clean'],  function () {
    return gulp.src(['src/assets/**/*'])
        .pipe(rev())
        .pipe(gulp.dest('dist/assets'))
        .pipe(rev.manifest('./dist/rev-manifest.json', {
            merge: true,
            base: './dist'
        })) // merge with the existing manifest (if one exists)
        .pipe(gulp.dest('dist/'));
});

gulp.task('clean', function () {
    return gulp.src('dist', {read: false})
        .pipe(clean());
});

We can now run the usemin task and all dependent tasks run automatically in their proper sequence.