How to efficiently manage file storage with Heroku

Facebooktwittergoogle_pluslinkedin

When using a cloud based platform to deploy an application, uploading static content is always an issue. These platforms have an ‘ephemeral file system’, so you can temporarily write files to disk. A common practice is to upload said content to specialized services like Amazon S3.

The pieces of code below can be easily applied to the “Nibs” Heroku app architecture. Heroku + Nibs – architecture for great customer-engagement apps (blog post).

How to efficiently manage file storage with Heroku

To make uploading pictures from Heroku really simple and fast, we’ll use S3. Here’s how to upload pictures from an Ionic app based on a NodeJS backend deployed on Heroku. The code example was taken from a Heroku project (https://github.com/heroku/nibs).

This can be applied to any kind of files, including large videos or pictures…

The backend
First thing first, when you want to upload a file to S3, you have to authorize your request using your AWS Access Key ID and a HMAC of the Policy object hashed with your AWS Secret (http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html).

Since you will be using confidential information (your AWS Secret) you have to generate the signature from your backend. 

We are using our S3 information (bucket name, access key and secret) to generate a SHA1 HMAC that we will return to our client app.

s3signing.js

var crypto = require('crypto'),
 config = require('./config'),
 winston = require('winston'),

bucket = config.s3.bucket,
 awsKey = config.s3.awsKey,
 secret = config.s3.secret;

exports.sign = function(req, res, next) {

winston.info('Signing S3 document');

var fileName = req.body.fileName,
 expiration = new Date(new Date().getTime() + 1000 * 60 * 5).toISOString();

var policy =
 { "expiration": expiration,
 "conditions": [
 {"bucket": bucket},
 {"key": fileName},
 {"acl": 'public-read'},
 ["starts-with", "$Content-Type", ""],
 ["content-length-range", 0, 524288000]
 ]};

policyBase64 = new Buffer(JSON.stringify(policy), 'utf8').toString('base64');
 signature = crypto.createHmac('sha1', secret).update(policyBase64).digest('base64');
 var response = {bucket: bucket, awsKey: awsKey, policy: policyBase64, signature: signature};
 winston.info(response);
 res.send({bucket: bucket, awsKey: awsKey, policy: policyBase64, signature: signature});

}

Client side
Now the client side of the upload :

Since this application is an Ionic one, we will put our code in an Angular module.

s3uploader.js

angular.module('nibs.s3uploader', [])

.factory('S3Uploader', function ($q, $window, $http, $ionicPopup, $rootScope) {

var signingURI = $rootScope.server.url + "/s3signing";

function upload(imageURI, fileName) {

console.log('Uploading ' + fileName + ' to S3');

var deferred = $q.defer(),
 ft = new FileTransfer(),
 options = new FileUploadOptions();

options.fileKey = "file";
 options.fileName = fileName;
 options.mimeType = "image/jpeg";
 options.chunkedMode = false;

console.log('Requesting signed doc ' + signingURI);
 $http.post(signingURI, {"fileName": fileName})
 .success(function (data) {
 console.log('Got signed doc: ' + JSON.stringify(data));
 options.params = {
 "key": fileName,
 "AWSAccessKeyId": data.awsKey,
 "acl": "public-read",
 "policy": data.policy,
 "signature": data.signature,
 "Content-Type": "image/jpeg"
 };

ft.upload(imageURI, "https://" + data.bucket + ".s3.amazonaws.com/",
 function (e) {
 console.log("Upload succeeded");
 deferred.resolve(e);
 },
 function (e) {
 $ionicPopup.alert({title: 'Oops', content: 'The image upload failed'});
 deferred.reject(e);
 }, options);

})
 .error(function (data, status, headers, config) {
 console.log(status);
 });

return deferred.promise;

}

return {
 upload: upload
 }

}); 

Directly upload –  without sending the actual file

We first request a signature from our backend by sending the fileName and then directly proceed to the upload to S3 without sending the actual file to our backend (saving resources and money).

As you can see, uploading pictures from a Heroku application is really simple and can be implemented in a matter of hours.

Pierre-Fraisse-SylpheoAbout Pierre Fraisse
Pierre Fraisse is Lead Heroku Developer at Sylpheo.

Facebooktwittergoogle_pluslinkedin