Rails Aws S3 Resource Multi Part Upload
Uploading Files to Amazon S3 With a Rails API Backend and Javascript Frontend
This guide will walk you through a method to integrate S3 hosting with Rails-as-an-API. I will also talk about how to integrate with the frontend. Notation while some of the setup is focused on Heroku, this is applicable for whatever Rails API backend. There are many short guides out there, but this is intended to bring everything together in a clear manner. I put troubleshooting tips at the end, for some of the errors I ran into.
For this guide, I had a Runway API app in one working directory, and a React app in a different directory. I will assume you already know the basics of connecting your frontend to your backend, and assume that you know how to run them locally. This guide is quite long, and may accept you lot a few hours to follow along with. Delight accept breaks.
Background
We will be uploading the file straight from the frontend. One reward of this is that information technology saves united states of america on big requests. If we uploaded to the backend, then had the backend transport information technology to S3, that will be two instances of a potentially large request. Some other advantage is because of Heroku'due south setup: Heroku has an "ephemeral filesystem." Your files may remain on the system briefly, simply they volition always disappear on a organization bike. You can effort to upload files to Heroku then immediately upload them to S3. Yet, if the filesystem cycles in that fourth dimension, y'all will upload an incomplete file. This is less relevant for smaller files, but nosotros will play it prophylactic for the purposes of this guide.
Our backend will serve ii roles: it will relieve metadata about the file, and handle all of the authentication steps that S3 requires. It will never touch the actual files.
The flow volition look similar this:
- The frontend sends a request to the Rails server for an authorized url to upload to.
- The server (using Active Storage) creates an authorized url for S3, then passes that back to the frontend.
- The frontend uploads the file to S3 using the authorized url.
- The frontend confirms the upload, and makes a request to the backend to create an object that tracks the needed metadata.
 |
| Steps i and ii are in diagram 2.1. Steps 3 and 4 are diagrams 2.2 and ii.iii, respectively. |
| Image taken from Applaudo Studios |
Setting upward S3
First, nosotros will fix the S3 resource we want. Create two S3 buckets, prod and dev. You can let everything be default, but take note of the bucket region. You lot volition need that afterwards.
 |
| What yous run into in S3 when making a new bucket. |
Next, nosotros will gear up Cross-Origin Resource Sharing (CORS). This will permit you lot to make Mail service & PUT requests to your saucepan. Become into each bucket, Permissions -> CORS Configuration. For at present, we will just use a default config that allows everything. We volition restrict it later.
<?xml version="ane.0" encoding="UTF-8"?> <CORSConfiguration xmlns= "http://s3.amazonaws.com/doc/2006-03-01/" > <CORSRule> <AllowedOrigin>*</AllowedOrigin> <AllowedMethod>GET</AllowedMethod> <AllowedMethod>POST</AllowedMethod> <AllowedMethod>PUT</AllowedMethod> <AllowedHeader>*</AllowedHeader> </CORSRule> </CORSConfiguration> Next, we will create some security credentials to allow our backend to practise fancy things with our bucket. Click the dropdown with your account proper noun, and select My Security Credentials. This volition take you lot to AWS IAM.
 |
| Accessing "My Security Credentials" |
Once in the Identity and Access Management console, you lot should go to the access keys section, and create a new access central.
 |
| Location of AWS access keys |
Here, it will create a primal for you. Information technology will never show you the hush-hush again, then make sure you save these values in a file on your computer.
Rails API Backend
Again, I assume you lot know how to create a basic Rails API. I will be attaching my file to a user model, but you lot tin can attach information technology to whatever you want.
Environment Variables
Add ii gems to your Gemfile: gem 'aws-sdk-s3' and gem 'dotenv-rails', then packet install. The first precious stone is the S3 software development kit. The 2d jewel allows Rails to use a .env file.
The access central and region (from AWS) are needed inside Rails. While locally developing, we will pass these values using a .env file. While on Heroku, we can set the values using heroku config, which we volition explore at the end of this guide. Nosotros will non be using a Procfile. Create the .env file at the root of your directory, and be sure to add it to your gitignore. You don't want your AWS account secrets ending upwards on Github. Your .env file should include:
AWS_ACCESS_KEY_ID=YOURACCESSKEY AWS_SECRET_ACCESS_KEY=sEcReTkEyInSpoNGeBoBCaSe S3_BUCKET=your-app-dev AWS_REGION=your-region-one Storage Setup
Run rails active_storage:install. Agile Storage is a library that helps with uploads to various deject storages. Running this command will create a migration for a table that will handle the files' metadata. Make sure to runway db:migrate.
Side by side, we volition modify the files that keep track of the Active Storage environment. There should be a config/storage.yml file. We will add an amazon S3 storage pick. Its values come from our .env file.
amazon : service : S3 access_key_id : <%= ENV['AWS_ACCESS_KEY_ID'] %> secret_access_key : <%= ENV['AWS_SECRET_ACCESS_KEY'] %> region : <%= ENV['AWS_REGION'] %> bucket : <%= ENV['S3_BUCKET'] %> Adjacent, go to config/enviroments, and update your production.rb and development.rb. For both of these, modify the Active Storage service to your newly added i:
config . active_storage . service = :amazon Finally, nosotros demand an initializer for the AWS S3 service, to set it upward with the access primal. Create a config/initializers/aws.rb, and insert the following code:
require 'aws-sdk-s3' Aws . config . update ({ region: ENV [ 'AWS_REGION' ], credentials: Aws :: Credentials . new ( ENV [ 'AWS_ACCESS_KEY_ID' ], ENV [ 'AWS_SECRET_ACCESS_KEY' ]), }) S3_BUCKET = Aws :: S3 :: Resource . new . bucket ( ENV [ 'S3_BUCKET' ]) Nosotros are now ready to store files. Adjacent nosotros will talk about the Runway model and controller setup.
Model
For my app, I am uploading user resumes, for the user model. You may be uploading images or other files. Experience free to change the variable names to whatever yous like.
In my user.rb model file, we need to adhere the file to the model. We will also create a helper method that shares the file'southward public URL, which will get relevant later.
class User < ApplicationRecord has_one_attached :resume def resume_url if resume . attached? resume . blob . service_url finish finish end Make sure that the model does not have a corresponding column in its table. There should exist no resume cavalcade in my user's schema.
Straight Upload Controller
Next nosotros will create a controller to handle the hallmark with S3 through Active Storage. This controller will await a Mail service request, and will render an object that includes a signed url for the frontend to PUT to. Run rails grand controller direct_upload to create this file. Additionally, add a route to routes.rb:
mail '/presigned_url' , to: 'direct_upload#create' The contents of the direct_upload_controller.rb file tin can be institute here.
The actual magic is handled by the ActiveStorage::Hulk.create_before_direct_upload! function. Everything else just formats the input or output a little bit. Take a look at blob_params; our frontend will be responsible for determining those.
Testing
At this point, information technology might be useful to verify that the endpoint is working. You can examination this functionality with something similar curl or Postman. I used Postman.
Run your local server with rails s, and so you can test your direct_upload#create endpoint by sending a POST request. There are a few things you will need:
- On a Unix machine, you tin get the size of a file using
ls -l. - If you lot take a different type of file, make sure to change the
content_typevalue. - S3 also expects a "checksum", so that it can verify that information technology received an uncorrupted file. This should be the MD5 hash of the file, encoded in base64. You can become this by running
openssl md5 -binary filename | base64.
Your POST request to /presigned_url might look like this:
{ "file" : { "filename" : "test_upload" , "byte_size" : 67969 , "checksum" : "VtVrTvbyW7L2DOsRBsh0UQ==" , "content_type" : "application/pdf" , "metadata" : { "bulletin" : "active_storage_test" } } } The response should have a pre-signed URL and an id:
{ "direct_upload" : { "url" : "https://your-s3-bucket-dev.s3.amazonaws.com/uploads/uuid?some-really-long-parameters" , "headers" : { "Content-Blazon" : "awarding/pdf" , "Content-MD5" : "VtVrTvbyW7L2DOsRBsh0UQ==" } }, "blob_signed_id" : "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBSQT09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--8a8b5467554825da176aa8bca80cc46c75459131" } The response direct_upload.url should take several parameters attached to information technology. Don't worry too much well-nigh it; if in that location was something wrong you would just get an error.
| Ten-Amz-Algorithm |
| Ten-Amz-Credential |
| X-Amz-Date |
| Ten-Amz-Expires |
| X-Amz-SignedHeaders |
| X-Amz-Signature |
Your directly upload now has an expiration of 10 minutes. If this looks correct, we tin use the direct_upload object to make a PUT asking to S3. Employ the same url, and make sure you include the headers. The trunk of the request will be the file you are looking to include.
 |
| What the PUT looks similar in Postman. Headers non shown. |
You lot should get a unproblematic empty response with a code of 200. If you go to the S3 bucket in the AWS panel, yous should see the folder and the file. Note that you can't really view the file (you can only view its metadata). If you lot try to click the "Object URL", it will tell you Access Denied. This is okay! Nosotros don't have permission to read the file. Earlier, in my user.rb model, I put a helper function that uses Agile Storage to go a public URL. We will accept a look at that in a bit.
 |
| The uploaded file |
User Controller
If you recall our flow:
- The frontend sends a request to the server for an authorized url to upload to.
-
The server (using Active Storage) creates an authorized url for S3, and so passes that back to the frontend.Washed. - The frontend uploads the file to S3 using the authorized url.
- The frontend confirms the upload, and makes a asking to the backend to create an object that tracks the needed metadata.
The backend still needs ane flake of functionality. It needs to exist able to create a new record using the uploaded file. For example, I am using resume files, and attaching them to users. For a new user cosmos, it expects a first_name, last_name, and email. The resume will take the form of signed_blob_id nosotros saw before. Active Storage simply needs this ID to connect the file to your model example. Hither is what my users_controller#create looks similar, and I also fabricated a gist:
def create resume = params [ :pdf ] params = user_params . except ( :pdf ) user = User . create! ( params ) user . resume . attach ( resume ) if resume . nowadays? && !! user render json: user . as_json ( root: false , methods: :resume_url ). except ( 'updated_at' ) end private def user_params params . allow ( :electronic mail , :first_name , :last_name , :pdf ) end The biggest new thing is the resume.attach phone call. Also note that we are returning the json of the user, and including our created resume_url method. This is what allows united states of america to view the resume.
Your params may look different if your model is unlike. We can once again test this with Postman or coil. Here is a json Post asking that I would brand to the /users endpoint:
{ "email" : "test08@email1.com" , "first_name" : "Test" , "last_name" : "er" , "pdf" : "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBLdz09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3fe2ec7e27bb9b5678dd9f4c7786032897d9511b" } This is much like a normal user creation, except we phone call attach on the file ID that is passed with the asking. The ID is from the response of our first request, the blob_signed_id field. Y'all should get a response that represents the user, but has a resume_url field. You can follow this public url to see your uploaded file! This url comes from the blob.service_url we included in the user.rb model.
 |
| The response, containing the newly created user. |
If this is all working, your backend is probably all set.
The Javascript Frontend
Remember our overall request flow. If nosotros simply consider the requests that the frontend performs, it will look like this:
- Make POST asking for signed url.
- Brand PUT request to S3 to upload the file.
- Brand Mail to
/usersto create new user.
We have already tested all of this using curl/Postman. At present it just needs to be implemented on the frontend. I am also going to presume you know how to get a file into Javascript from a computer. <input> is the simplest method, but there are plenty of guides out there.
The only hard office of this is computing the checksum of the file. This is a little weird to follow, and I had to judge-and-check my manner through a flake of this. To start with, we volition npm install crypto-js. Crypto JS is a cryptographic library for Javascript.
Note: if you lot are using vanilla Javascript and can't use npm, here are some directions to import it with a CDN. You will need:
-
rollups/md5.js -
components/lib-typedarrays-min.js -
components/enc-base64-min.js
Then, nosotros will read the file with FileReader earlier hashing it, co-ordinate to the following code. Hither is a link to the respective gist.
import CryptoJS from ' crypto-js ' // Note that for larger files, you may want to hash them incrementally. // Taken from https://stackoverflow.com/questions/768268/ const md5FromFile = ( file ) => { // FileReader is upshot driven, does not return promise // Wrap with promise api so we can telephone call w/ async wait // https://stackoverflow.com/questions/34495796 return new Hope (( resolve , pass up ) => { const reader = new FileReader () reader . onload = ( fileEvent ) => { let binary = CryptoJS . lib . WordArray . create ( fileEvent . target . result ) const md5 = CryptoJS . MD5 ( binary ) resolve ( md5 ) } reader . onerror = () => { pass up ( ' oops, something went wrong with the file reader. ' ) } // For some reason, readAsBinaryString(file) does not work correctly, // and then we will handle it as a word array reader . readAsArrayBuffer ( file ) }) } export const fileChecksum = async ( file ) => { const md5 = await md5FromFile ( file ) const checksum = md5 . toString ( CryptoJS . enc . Base64 ) return checksum } At the end of this, we will accept an MD5 hash, encoded in base64 (merely similar we did above with the terminal). Nosotros are almost washed! The merely matter we need are the bodily requests. I will paste the code, but here is a link to a gist of the JS request lawmaking.
import { fileChecksum } from ' utils/checksum ' const createPresignedUrl = async ( file , byte_size , checksum ) => { allow options = { method : ' Mail service ' , headers : { ' Accept ' : ' application/json ' , ' Content-Type ' : ' application/json ' , }, torso : JSON . stringify ({ file : { filename : file . name , byte_size : byte_size , checksum : checksum , content_type : ' application/pdf ' , metadata : { ' message ' : ' resume for parsing ' } } }) } permit res = expect fetch ( PRESIGNED_URL_API_ENDPOINT , options ) if ( res . status !== 200 ) return res return await res . json () } export const createUser = async ( userInfo ) => { const { pdf , email , first_name , last_name } = userInfo // To upload pdf file to S3, we need to do iii steps: // one) request a pre-signed PUT request (for S3) from the backend const checksum = await fileChecksum ( pdf ) const presignedFileParams = await createPresignedUrl ( pdf , pdf . size , checksum ) // two) send file to said PUT request (to S3) const s3PutOptions = { method : ' PUT ' , headers : presignedFileParams . direct_upload . headers , body : pdf , } permit awsRes = await fetch ( presignedFileParams . direct_upload . url , s3PutOptions ) if ( awsRes . status !== 200 ) return awsRes // 3) confirm & create user with backend let usersPostOptions = { method : ' POST ' , headers : { ' Have ' : ' application/json ' , ' Content-Blazon ' : ' application/json ' }, trunk : JSON . stringify ({ email : electronic mail , first_name : first_name , last_name : last_name , pdf : presignedFileParams . blob_signed_id , }) } let res = await fetch ( USERS_API_ENDPOINT , usersPostOptions ) if ( res . status !== 200 ) return res render wait res . json () } Note that you lot need to provide the two global variables: USERS_API_ENDPOINT and PRESIGNED_URL_API_ENDPOINT. Also annotation that the pdf variable is a Javascript file object. Again, if you are not uploading pdfs, be certain to alter the appropriate content_type.
You at present accept the required Javascript to make your application work. Just attach the createUser method to course inputs, and brand sure that pdf is a file object. If y'all open up the Network tab in your browser devtools, you lot should see three requests fabricated when yous call the method: one to your API's presigned_url endpoint, ane to S3, and one to your API'due south user create endpoint. The final one will besides return a public URL for the file, so you can view it for a express time.
Terminal Steps and Cleanup
S3 Buckets
Brand certain your prod app is using a different bucket from your development. This is then y'all tin can restrict its CORS policy. It should only accept PUT requests from one source: your production frontend. For case, here is my product CORS policy:
<?xml version="1.0" encoding="UTF-8"?> <CORSConfiguration xmlns= "http://s3.amazonaws.com/doc/2006-03-01/" > <CORSRule> <AllowedOrigin>https://myfrontend.herokuapp.com</AllowedOrigin> <AllowedMethod>Post</AllowedMethod> <AllowedMethod>PUT</AllowedMethod> <AllowedMethod>GET</AllowedMethod> <AllowedHeader>*</AllowedHeader> </CORSRule> </CORSConfiguration> Yous don't need to enable CORS for the communication between Rails and S3, because that is not technically a request, it is Agile Storage.
Heroku Production Settings
You may accept to update your Heroku prod surroundings. After you push your code, don't forget to heroku run rails db:drift. You will as well demand to make certain your environment variables are correct. You can view them with heroku config. You can ready them by going to the app's settings in the Heroku dashboard. You lot can likewise set them with heroku config:set AWS_ACCESS_KEY_ID=thirty AWS_SECRET_ACCESS_KEY=yyy S3_BUCKET=bucket-for-app AWS_REGION=my-region-ane.
Public Viewing of Files
The public URL you receive to view the files is temporary. If y'all want your files to be permanently publicly viewable, you will need to have a few more steps. That is outside the realm of this guide.
Some Troubleshooting
Here are some errors I ran into while building this guide. It is not comprehensive, but may assist you.
Problems with server initialization: brand certain the names in your .env files match the names where you lot access them.
Error: missing host to link to for the get-go asking. In my case, this meant I had non put :amazon equally my Agile Storage source in development.rb.
StackLevelTooDeep for last asking. I had this result when calling users_controller#create because I had not removed the "resume" field from my schema. Make certain your database schema does not include the file. That should only be referenced in the model with has_one_attached.
AWS requests fail after changing CORS: make sure there are no abaft slashes in your URL within the CORS XML.
Debugging your checksum: this is a hard one. If you are getting an error from S3 saying that the computed checksum is not what they expected, this ways there is something wrong with your adding, and therefore something incorrect with the Javascript you received from hither. If you double check the code you copied from me and can't find a departure, yous may have to figure this out on your ain. For Javascript, you can cheque the MD5 value by calling .toString() on it with no arguments. On the control line, you can drop the --binary flag.
Sources and References
Much of this was taken from Arely Viana'southward blog post for Applaudo Studios. I linked the code together, and figured out how the frontend would look. A huge shout-out to them!
Hither are some other resources I plant useful:
- Heroku's guide for S3 with Rails - this is not for Rail equally an API, but it does talk about surround setup
- The code for Arely'south guide - also has some instance JSON requests
- Rails Active Storage Overview
- Uploading to S3 with JS - this also uses AWS Lambda, with no backend
Source: https://elliott-king.github.io/2020/09/s3-heroku-rails/
Post a Comment for "Rails Aws S3 Resource Multi Part Upload"