API with NestJS #10. Uploading public files to Amazon S3

• 1. API with NestJS #1. Controllers, routing and the module structure
• 2. API with NestJS #2. Setting up a PostgreSQL database with TypeORM
• 3. API with NestJS #3. Authenticating users with bcrypt, Passport, JWT, and cookies
• 4. API with NestJS #4. Error handling and data validation
• 5. API with NestJS #5. Serializing the response with interceptors
• 6. API with NestJS #6. Looking into dependency injection and modules
• 7. API with NestJS #7. Creating relationships with Postgres and TypeORM
• 8. API with NestJS #8. Writing unit tests
• 9. API with NestJS #9. Testing services and controllers with integration tests
• 10. API with NestJS #10. Uploading public files to Amazon S3

While storing files directly in the database is doable, it might not be the best approach. Files can take a lot of space, and it might impact the performance of the application. Also, it increases the size of the database and, therefore, makes backups bigger and slower. A good alternative is storing files separately using an external provider, such as Google Cloud, Azure, or Amazon AWS.

In this article, we look into uploading files to Amazon Simple Storage Service, also referred to as S3 . You can find all the code from this series in this repository .

Connecting to Amazon S3

Amazon S3 provides storage that we can use with any type of file. We organize files into buckets and manage them in our API through an SDK.

Once we create the AWS account, we can log in as a root user. Even though we might authorize as root to use the S3 through our API, this is not the best approach.

Setting up a user

Let’s create a new user that has a restricted set of permissions. To do so, we need to open the Identity and Access Management (IAM) panel and create a user:

Since we want this user to be able to manage everything connected to S3, let’s set up proper access.

After doing that, we are presented with Access key ID and  Secret access key . We need those to connect to AWS through our API. We also need to choose one of the available regions .

Let’s add them to our . env file:

.env

# ...
AWS_REGION=eu-central-1
AWS_ACCESS_KEY_ID=*******
AWS_SECRET_ACCESS_KEY=*******

Also, let’s add it to our environment variables validation schema in AppModule:

ConfigModule.forRoot({
  validationSchema: Joi.object({
    POSTGRES_HOST: Joi.string().required(),
    POSTGRES_PORT: Joi.number().required(),
    POSTGRES_USER: Joi.string().required(),
    POSTGRES_PASSWORD: Joi.string().required(),
    POSTGRES_DB: Joi.string().required(),
    JWT_SECRET: Joi.string().required(),
    JWT_EXPIRATION_TIME: Joi.string().required(),
    AWS_REGION: Joi.string().required(),
    AWS_ACCESS_KEY_ID: Joi.string().required(),
    AWS_SECRET_ACCESS_KEY: Joi.string().required(),
    PORT: Joi.number(),
  })
}),

Connecting to AWS through SDK

Once we have the necessary variables, we can connect to AWS using the official SDK for Node. Let’s install it first.

npm install aws-sdk @types/aws-sdk

Since we’ve got everything that we need to configure the SDK, let’s use it. One of the ways to do so is to use aws - sdk straight in our  main . ts file.

main.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import * as cookieParser from 'cookie-parser';
import { ValidationPipe } from '@nestjs/common';
import { ExcludeNullInterceptor } from './utils/excludeNull.interceptor';
import { ConfigService } from '@nestjs/config';
import { config } from 'aws-sdk';
 
async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  app.useGlobalPipes(new ValidationPipe());
  app.useGlobalInterceptors(new ExcludeNullInterceptor());
  app.use(cookieParser());
 
  const configService = app.get(ConfigService);
  config.update({
    accessKeyId: configService.get('AWS_ACCESS_KEY_ID'),
    secretAccessKey: configService.get('AWS_SECRET_ACCESS_KEY'),
    region: configService.get('AWS_REGION'),
  });
 
  await app.listen(3000);
}
bootstrap();

Creating our first bucket

In Amazon S3 data is organized in buckets. We can have multiple buckets with different settings.

Let’s open the Amazon S3 panel and create a bucket. Please note that the name of the bucket must be unique.

We can set up our bucket to contain public files . All files that we upload to this bucket will be publicly available. We might use it to manage files such as avatars.

The last step here is to add the name of the bucket to our environment variables.

.env

# ...
AWS_PUBLIC_BUCKET_NAME=nestjs-series-public-bucket

src/app.module.ts

ConfigModule.forRoot({
  validationSchema: Joi.object({
    // ...
    AWS_PUBLIC_BUCKET_NAME: Joi.string().required(),
  })
}),

Uploading images through our API

Since we’ve got the AWS connection set up, we can proceed with uploading our files. For starters, let’s create a PublicFile entity.

src/files/publicFile.entity.ts

import { Column, Entity, PrimaryGeneratedColumn } from 'typeorm';
 
@Entity()
class PublicFile {
  @PrimaryGeneratedColumn()
  public id: number;
 
  @Column()
  public url: string;
 
  @Column()
  public key: string;
}
 
export default PublicFile;

By saving the URL directly in the database, we can access the public file very quickly. The key property uniquely identifies the file in the bucket. We need it to access the file, for example, if we want to delete it.

The next step is creating a service that uploads files to the bucket and saves the data about the file to our Postgres database. Since we want keys to be unique, we use the uuid library :

npm install uuid @types/uuid

src/files/files.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import PublicFile from './publicFile.entity';
import { S3 } from 'aws-sdk';
import { ConfigService } from '@nestjs/config';
import { v4 as uuid } from 'uuid';
 
@Injectable()
export class FilesService {
  constructor(
    @InjectRepository(PublicFile)
    private publicFilesRepository: Repository<PublicFile>,
    private readonly configService: ConfigService
  ) {}
 
  async uploadPublicFile(dataBuffer: Buffer, filename: string) {
    const s3 = new S3();
    const uploadResult = await s3.upload({
      Bucket: this.configService.get('AWS_PUBLIC_BUCKET_NAME'),
      Body: dataBuffer,
      Key: `${uuid()}-${filename}`
    })
      .promise();
 
    const newFile = this.publicFilesRepository.create({
      key: uploadResult.Key,
      url: uploadResult.Location
    });
    await this.publicFilesRepository.save(newFile);
    return newFile;
  }
}

The uploadPublicFile method expects a buffer . It is a chunk of memory that keeps a binary representation of our file. If you want to know more about it, check out  Node.js TypeScript #3. Explaining the Buffer .

Creating an endpoint for uploading files

Now we need to create an endpoint for the user to upload the avatar. To link the files with users, we need to modify the UserEntity by adding the  avatar column.

src/users/user.entity.ts

import { Entity, JoinColumn, OneToOne } from 'typeorm';
import PublicFile from '../files/publicFile.entity';
 
@Entity()
class User {
  // ...
 
  @JoinColumn()
  @OneToOne(
    () => PublicFile,
    {
      eager: true,
      nullable: true
    }
  )
  public avatar?: PublicFile;
}
 
export default User;

If you wan to know more about relationships with Postgres and TypeORM, check out API with NestJS #7. Creating relationships with Postgres and TypeORM

Let’s add a method to the UsersService that uploads files and links them to the user.

src/users/users.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import User from './user.entity';
import { FilesService } from '../files/files.service';
 
@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepository: Repository<User>,
    private readonly filesService: FilesService
  ) {}
 
  // ...
 
  async getById(id: number) {
    const user = await this.usersRepository.findOne({ id });
    if (user) {
      return user;
    }
    throw new HttpException('User with this id does not exist', HttpStatus.NOT_FOUND);
  }
 
  async addAvatar(userId: number, imageBuffer: Buffer, filename: string) {
    const avatar = await this.filesService.uploadPublicFile(imageBuffer, filename);
    const user = await this.getById(userId);
    await this.usersRepository.update(userId, {
      ...user,
      avatar
    });
    return avatar;
  }
}

This might be a fitting place to include some functionalities like checking the size of the image or compressing it.

The last piece is adding an endpoint the users can send avatars to. To do that, we follow the NestJS documentation and use the  FileInterceptor that utilizes multer under the hood.

src/users/users.controller.ts

import { UsersService } from './users.service';
import { Controller, Post, Req, UploadedFile, UseGuards, UseInterceptors } from '@nestjs/common';
import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard';
import RequestWithUser from '../authentication/requestWithUser.interface';
import { FileInterceptor } from '@nestjs/platform-express';
import { Express } from 'express';
 
@Controller('users')
export class UsersController {
  constructor(
    private readonly usersService: UsersService,
  ) {}
 
  @Post('avatar')
  @UseGuards(JwtAuthenticationGuard)
  @UseInterceptors(FileInterceptor('file'))
  async addAvatar(@Req() request: RequestWithUser, @UploadedFile() file: Express.Multer.File) {
    return this.usersService.addAvatar(request.user.id, file.buffer, file.originalname);
  }
}

The file above has quite a few useful properties such as the mimetype . You can use it if you want some additional validation and disallow certain types of files.

Deleting existing files

Aside from uploading files, we also need a way to remove them. To keep our database consistent with the Amazon S3 storage, we remove the files from both places. First, let’s add the method to the FilesService .

src/files/files.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import PublicFile from './publicFile.entity';
import { S3 } from 'aws-sdk';
import { ConfigService } from '@nestjs/config';
 
@Injectable()
export class FilesService {
  constructor(
    @InjectRepository(PublicFile)
    private publicFilesRepository: Repository<PublicFile>,
    private readonly configService: ConfigService
  ) {}
 
  // ...
 
  async deletePublicFile(fileId: number) {
    const file = await this.publicFilesRepository.findOne({ id: fileId });
    const s3 = new S3();
    await s3.deleteObject({
      Bucket: this.configService.get('AWS_PUBLIC_BUCKET_NAME'),
      Key: file.key,
    }).promise();
    await this.publicFilesRepository.delete(fileId);
  }
}

Now, we need to use it in our UsersService . Important addition is that when a user uploads an avatar while already having one, we delete the old one.

src/users/users.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import User from './user.entity';
import { FilesService } from '../files/files.service';
 
@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepository: Repository<User>,
    private readonly filesService: FilesService
  ) {}
 
  // ...
 
  async addAvatar(userId: number, imageBuffer: Buffer, filename: string) {
    const user = await this.getById(userId);
    if (user.avatar) {
      await this.usersRepository.update(userId, {
        ...user,
        avatar: null
      });
      await this.filesService.deletePublicFile(user.avatar.id);
    }
    const avatar = await this.filesService.uploadPublicFile(imageBuffer, filename);
    await this.usersRepository.update(userId, {
      ...user,
      avatar
    });
    return avatar;
  }
 
  async deleteAvatar(userId: number) {
    const user = await this.getById(userId);
    const fileId = user.avatar?.id;
    if (fileId) {
      await this.usersRepository.update(userId, {
        ...user,
        avatar: null
      });
      await this.filesService.deletePublicFile(fileId)
    }
  }
}

Summary

In this article, we’ve learned the basics of how Amazon S3 works and how to use it in our API. To do that, we’ve provided the necessary credentials to AWS SDK. Thanks to that, we were able to upload and delete files to AWS. We’ve also kept our database in sync with Amazon S3, to track our files. To upload files through API we’ve used the FileInterceptor, which uses Multer under the hood.

Since there is more to Amazon S3 than handling public files, there is still quite a bit to cover here, and you might expect it in this series.

Series Navigation

<< API with NestJS #9. Testing services and controllers with integration tests