Another 'not in the series' post...
So I came across an interesting error with AWS S3 and an open source nodejs module where every time we tried to save a file into a S3 bucket we would get an error with the key word SignatureDoesNotMatch.
After a lot of trolling the interwebs for a solution, the most common answer I found was that the time on my server must be well off (as in days or more) the time AWS S3 uses. Lets just say this isn't the issue, and probably is never the issue given the 'Signature' AWS are checking isn't time based unless you try to roll your own S3 module (again, not the issue in this case).
Another answer that I kept coming across has to do with the 'x-amz-acl' header value when it is set to 'public-read'. This is also nothing to do with the error we came across.
One last answer I feel is worth mentioning that I also kept coming across was regarding the AWS Secret Access Key. If (again in a self-rolled module) you don't encode the Secret correctly to cater for various characters such as + or /, then you can also screw up the Signature. To start with I thought this was a real issue as AWS did generate a Secret with a special character in it, but after switching out the Access Key/Secret Access Key it turned out that was not the issue in this particular case even though one of the dependancies being used was indeed a self-rolled S3 module.
So, the error message (found in the logging we have on the S3 bucket) looked similar to this:
33a644d4f8284a5fdc2dea794a4bd26e4fc4b22d91a8ec7a4b2122044e935e23 s3bucketname [19/Jan/2015:02:01:34 +0000] 1.2.3.4 - FCEA0D5CBAF74C19 REST.PUT.OBJECT filename.png "PUT /filename.png HTTP/1.1" 403 SignatureDoesNotMatch 907 - 9 - "-" "-" -
The headers for authorisation being sent in the request going up to AWS looked like this (bucket, access key and secret all fake):
{ 'content-length': 33063,
'content-type': undefined,
'x-amz-acl': 'public-read',
date: 'Mon, 19 Jan 2015 02:01:34 GMT',
host: 's3bucketname.s3.amazonaws.com',
authorization: 'AWS AKIAIC7S2BCE4G26XJ1A:46sAv23fbalV6xTADjO2SQKlsavfWf8aFRA5L3TXw=' }
After adding in some more debugging into the nodejs module as it was not returning an error, we found it was receiving this response body back from AWS (AWS Access Key and data etc is still fake of course):
<?xml version="1.0" encoding="UTF-8"?>
<Error>{gfm-js-extract-pre-1}<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAIC7S2BCE4G26XJ1A</AWSAccessKeyId><StringToSign>PUT
undefined
Mon, 19 Jan 2015 02:01:34 GMT
x-amz-acl:public-read
/s3bucketname/filename.png</StringToSign><SignatureProvided>46sAv23fbalV6xTADjO2SQKlsavfWf8aFRA5L3TXw=</SignatureProvided><StringToSignBytes>50 55 54 ...</StringToSignBytes><RequestId>DC8E1ADBA8D67C4B</RequestId><HostId>uvTaUv023Fhlnp5dwr+8LR7MfSeIBFavaulaJq3xkhMHYH3hfKLZ33iD7zm35BxZCBi4lawrGrh</HostId></Error>
The key word in the response body is 'undefined'. What does that mean exactly? Well, it means that something in the underlying submission to the S3 bucket was not defined. Duh. In this case ContentType was not set, and when AWS runs their crypto checks, the 'undefined' ContentType was breaking their signature check as it isn't catered for and therefore gives out an error.
On a side note, I do wish more open source developers would put better error checking in - or at least comprehensive debug logging.... Some unit tests would be nice too while I'm wishing ;)
Failing on an undefined and expected value is actually a good thing - AWS have no reason to accept an undefined ContentType, and it should always be sent as per their API documentation.
In the end it was obvious based on the headers being sent that ContentType was the culprit, and sure enough in the open source module they had assumed that there is always a 'mimetype' sub-object available, whereas some combinations of OS and browser sometimes just call that sub-object 'type'. Needless to say once ContentType had an actual value, everything started working again.
For reference, the signature that AWS is checking uses this technique:
First, compile a string with the relevant information in it:
stringtosign = header info such as content-type, filename, other file info...
Then, create a HMAC signature using the AWS Secret Access Key (a form of password):
crypto.hmac(credentials.secretAccessKey, stringtosign, 'base64', 'sha1');
If you want to see the full code, you can view Amazon's own s3.js file here: https://github.com/aws/aws-sdk-js/blob/master/lib/signers/s3.js
Yes there is some information about the file that contains a date by default, and this doesn't have to be passed - it does have to be defined at least as an empty string (Amazon will then use their current date/time for the uploaded file at that point), but the date itself is not used as part of the HMAC math, so clock-skews are not an issue as mentioned above.
Hopefully this will help anybody else that comes across this issue in the future - look for an object that you are not defining but trying to send, fix it, and all will be well.