How I Built a Deep Learning Powered Emoji Slackbot

How I Built a Deep Learning Powered Emoji Slackbot :smile: :confounded: :grinning:

And Why the Hard Part of AI has Nothing to Do with AI

As I put the finishing touches on my PhD thesis in computer systems at UC Berkeley, I often wonder what life in the AI world is like. My AI friends are constantly bragging about how deep learning will revolutionize everything from medicine to online shopping — and their papers get 100s of citations as soon as they’re posted (what the heck!). But I’ve always wondered how they actually use AI to solve real problems.

At the same time, I’ve noticed recently that I spend most of my time responding to Slack conversations. Inspired by a recent blog post (and experiencing much thesis writing apathy), I decided to build and deploy a deep learning powered Slackbot to automatically add reactions to Slack posts with the most appropriate emoji. :smiley: :+1:

In this post, I walk through my experience building this Slackbot powered by DeepMoji . I ended up deploying DeepMoji using two different frameworks (AWS SageMaker and Algorithmia ). Here, I compare my experience using both systems, highlighting some of the strengths, limitations, and opportunities for improvement in prediction serving systems. This post is pretty high-level but this repository has links with detailed instructions and code to reproduce my Slackbot.

Personally, I’ve spent the last few years working on scalable cloud systems. I was particularly excited to jump into this task because we’ve spent some time recently thinking about scalable prediction infrastructure . That said, I’m definitely not an expert on AI systems, so let me know if I’ve missed something here.

Step 0: Setting up a Slackbot

Setting up a Slackbot is fairly straightforward but involves a lot of clicking around on the Slack website. I documented the process here with some screen grabs, but at a high level, I created a bot, linked it to my workspace, and gave it permission to view messages and add emoji reactions.

Next step: pick a machine learning model that powers my Slackbot to react text with emoji!

Step 1: Get a model running on my laptop

So what AI magic should I invoke? I don’t have the data to train a model, but fortunately, there are tons of pre-trained models that I can repurpose. My AI colleagues tell me that this is an emerging trend in model design: start with an existing model and fine tune it for your task. In my case, the DeepMoji model does exactly what I need and is already trained and open-source.

For a little bit of context, DeepMoji is trained on 1.2 billion tweets with emojis to understand language sentiment and how it correlates with emoji. My use case is pretty simple: Send the model a Slack message and react with the emoji that the model believes best reflects the sentiment of the message.

Bad news: DeepMoji uses Python 2 (yuck). Good news: The good people at HugginFace developed torchMoji , a PyTorch implementation of DeepMoji that supports Python3. The torchMoji codebase had a bug that I had to fix to correctly parse the PackedSequence PyTorch object. The working version is available on my fork here . This part just requires a laptop, so you can follow along at home if you’re interested.

With the bug fixed, getting the trained model is easy . First, I installed torchMoji as a package:

pip3 install git+https://github.com/cw75/torchMoji

I needed the model weights and vocabulary file to initialize the model. The vocabulary file was already in the repo ( here ). To download the weights, I ran:

python3 scripts/download_weights.py

Now that all the prerequisites are downloaded, making a prediction takes only a few lines of Python:

You can test it to make sure everything works:

Pretty cool! I didn’t even need to get a GPU.

Next step: turn this into a prediction service with a REST web interface that my Slackbot can invoke!

Step 2: Set up a prediction service

Serving predictions is the systems challenge in this scenario; the operational engine that will make this bot actually work. The important question is where should the prediction service live?

A strawman solution is to just wrap the prediction script I wrote above in a Flask server and use a tool like ngrok to generate a REST endpoint on my laptop or an EC2 instance. The problem is that I want to deploy the bot onto my lab’s Slack team so that everybody can play with it, so the bot needs to be always-on and scalable.

So I needed something real, and as a systems hacker, part of me wanted to build the service from scratch: pack my script into a Docker image, launch a Kubernetes cluster, and expose the REST endpoint via an AWS ELB .

On the other hand, as a researcher, I don’t want to reinvent the wheel, and I know cloud providers like Google, Amazon, and Microsoft have offerings for prediction serving. The first platform that came to mind was AWS SageMaker, a managed service on AWS that covers the entire machine learning lifecycle from model development to deployment. The team released a paper on SageMaker at SIGMOD’20, and I’ve been using AWS services for my research for years, so I figured it must be reasonable. I hadn’t realized I was stepping into the abyss…

Platform 1: Amazon SageMaker

I expected deploying a model on SageMaker to be simple. I already had a list of Python dependencies and the prediction script that I showed above. All I wanted to do was upload the script, dependencies, and model to SageMaker, create a REST endpoint, and connect it to the Slackbot.

But alas, there’s no way to upload a model to SageMaker. Instead, I had to build a Docker image with all my Python dependencies installed. I also had to write a Flask server that wraps my prediction script and responds to SageMaker’s health check and model invocation requests — you can see the docs here , the Dockerfile here , and my Flask server here . It’s not all that complicated, but if I wasn’t familiar with Docker and Flask, this probably would’ve taken me a while to figure out.

As a systems student, I’ve been using Docker pretty regularly for years, but I’ve found that my friends in AI don’t know a lot about it. What do you think of Docker? Is building Docker containers really the state of the art in AI? Are people well-versed in it??

I am starting to wonder what SageMaker does for me. If I’m doing all this, why not just run my own Kubernetes cluster? It might even be simpler because Kubernetes automatically takes care of the health check for me!

SageMaker doesn’t work with DockerHub, so once I built the Docker image, I had to push it to Amazon’s own Docker registry — AWS ECR (Elastic Container Registry). Next, I created a SageMaker model that pulls this Docker image and a SageMaker “endpoint” that uses my model. This required selecting a node type ( ml.m4.large ) and the cluster size. Finally, I could create my REST endpoint!

After waiting 10 minutes for my service to start, I was thrilled to see the status change from “Creating” to “InService” with a green checkmark. :white_check_mark: Awesome! I copy-pasted the endpoint URL into my Slackbot configuration, ready to dazzle everyone in the lab. And… Slack spat out this error:

Your URL didn't respond with the value of the challenge parameter.

Time to troubleshoot SageMaker! I had to write a Python script using boto3 (with proper permissions) and make an InvokeEndpoint call to test the endpoint. Using this script to debug for a half hour, I found that the endpoint itself was actually working fine. The real problem is that SageMaker can only be accessed from within an AWS virtual private cluster and is not supposed to accept requests from the outside internet, so Slack couldn’t reach it!

The solution, according to this blog post , is to “simply” use AWS’ API Gateway service to accept traffic from Slack, route the request to an AWS Lambda function that authenticates the request, and then use Lambda to send the prediction query to SageMaker. What?!? This deserves an architecture diagram:

That is some insane redirection when all I want to do is make my model accessible from the web! Okay, so… time to write a Lambda function. I’m always excited for an opportunity to use serverless systems — it’s been the focus of most of my research after all — but this one was a little more complicated than I expected.

Actually, this was way more complicated than I expected. I had to respond to Slack’s “challenge request (lines 24–29), authenticate each request to ward off hackers (lines 31–37), and explicitly parse a text field from the input HTTP request (lines 38–43) all before I could actually call the SageMaker model (lines 47–52). Finally, on line 55, I used the Slack Python API client to add a reaction to the input message.

There are a few other things worth noting that I’ve skipped over so far:

1. I had to create an IAM role (AWS’ security and authentication service) with the sagemaker:InvokeEndpoint policy and assign it to the Lambda function so that it could query my SageMaker endpoint.
2. When using the Slack API client, I had to provide the bot’s OAuth Access Token (line 13) and signing secret (line 15) which are unique to each Slackbot.
3. I had to explicitly pack third-party Python dependencies (the Slack SDK) and the Lambda function script into a zip file and upload it to AWS Lambda. Details here .

Wait, so… really, what is SageMaker doing for me? I’ve built a Docker container for my model, effectively written a proxy service, manually authenticated each request, and also need to set up a separate REST endpoint. Maybe doing this by hand with EC2 and Kubernetes would’ve been easier after all?

The last step was to create an API Gateway that routes subscription messages from Slack to my Lambda function. Along the way, I fell into what I imagine are some common traps. For example, I forgot to check the Use Lambda Proxy integration box, so API Gateway stripped HTTP headers off each request when forwarding it to Lambda. This led to an authentication failure as Slack embeds its tokens inside the header. I’ve documented all of the (many) steps here so you won’t repeat my mistakes.

Anyway, this all took another hour to debug, but API Gateway finally generated a real REST endpoint. I pasted the endpoint into my Slackbot’s subscription, and it finally worked! I sent a message to my bot on the Slack workspace, and I got a reaction. :sunglasses:

Voilà! Took me 6 hours to get all the pieces together, but it’s finally working! Each request takes a few seconds, which is… well, not very interactive. Considering the traffic flow shown in the “architecture diagram” above, it’s not surprising. Most of these services were not optimized for latency.

All in all, deploying torchMoji on SageMaker was a pain . Worse, after all the effort, each request took more than a second! Not to mention cost — in addition to keeping the model server running all the time, we’re now paying AWS for each synchronous Lambda invocation. It’s definitely not going to be cheap if all 400+ users on the RISELab Slack start playing around with the bot.

Given that using SageMaker was no walk in the park, I was pretty discouraged about putting more time into competing services from Microsoft Azure and Google Cloud. But that could be my mistake — if you’ve had better experiences with those offerings, let me know in the comments. I’d love to learn more!

Instead, I decided to look into a startup I’d heard about called Algorithmia — it specializes in model deployment and management, so hopefully it’s easier to use…

Platform 2: Algorithmia

Following these docs , I first created a Git repo linked to Algorithmia and put a slightly modified version of my prediction script from earlier into the repo. The script is pretty similar to the one I used locally — no need to worry about setting up web servers and handling health checks. I just had to change a couple lines of basic Python to use Algorithmia’s API. Off to a good start!

The repo also needs a requirements.txt file that contains a list of Python dependencies. This is standard operating procedure for Python projects, so I was glad to see I didn’t need to do anything crazy.

After committing these files, all I had to do was to publish my model and tag it with a version number (e.g., 0.0.1 ). And just like that, the torchMoji model was running on Algorithmia’s platform — way easier than SageMaker!

Algorithmia’s dashboard has a neat testing feature where I could type in test sentences and get emoji outputs. This is really helpful: I didn’t need another script to invoke the model to make sure everything is working as expected like I did with SageMaker.

Algorithmia also generated copy-paste code snippets in multiple languages that I could use to invoke my model. The Python version looked like this:

Things were looking great thus far. But when it came to integrating with Slack, Algorithmia is no better than SageMaker. From these docs :

In order for your Slack App to connect to Algorithmia, you’ll need an intermediate function: and API endpoint which can accept a GET request from Slack, validate the content, and then POST to one of Algorithmia’s APIs.

I still needed the Lambda glue code from earlier, which was a pain to write. The good news for me was that I could reuse most of my previous code for the Lambda function. I only needed to replace the SageMaker endpoint invocation with a call to Algorithmia, so — given that I’d already burned a few hours debugging this the first time around — doing it again wasn’t all that bad. :smiley: The full script is available here .

Still, what felt weird to me is that Algorithmia is supposed to be a serverless model serving platform . But to integrate with my Slackbot, I had to use another serverless platform (AWS Lambda) to route traffic and perform security checks. The end-to-end latency remained in single-digit seconds, as the requests still had to go through three hops: from Slack to API Gateway, from API Gateway to Lambda, and from Lambda to Algorithmia.

I can’t help but wonder: What’s the point of putting my model into a box — be it SageMaker or Algorithmia — if it remains so painful to connect it with other services like Slack. If “connecting” means I had to learn about, configure, and pay for two other services, wouldn’t it have been easier if I just put the prediction code into my Lambda function and skipped the special-purpose services entirely? It’s tough for me to believe that this is really the state of the art!

Reproducing the Slackbot

If you want to reproduce my Slackbot, all the code with detailed instructions is in this repository. I shared this repository with a few PhD students in the RISELab and all of them were able to reproduce and deploy the bot within an hour! However, they didn’t seem to have a super pleasant experience... Here is what they said:

Why does deploying a model require stitching together multiple services and learning their jargons? If the model can be run locally as simple as invoking a python script, the added complexity to deploy it to the cloud should be minimal.

— Zongheng Yang, a fourth-year PhD student working on machine learning for systems.

Following the instructions to get this simple bot set up was an exercise in blindly being led by an expert through the dark; the sheer number of ways to get something wrong makes diagnosing misconfiguration errors essentially infeasible for a newcomer.

— an anonymous third-year PhD student working on systems for machine learning.

Summing Up

We all know that AI infrastructure and areas like MLOps are super hot right now. It’s all everyone’s talking about. But in reality, there’s a lot of work that needs to be done to meet even the basic needs of online prediction serving. My ambitions here were modest — a simple model, with limited scalability. Getting that far was annoying, and performance was terrible.

All that said, I’m sure there are other prediction serving systems out there. We ourselves just built a prediction pipeline DSL on top of a serverless platform we’ve worked on in our lab. If you know of production systems or techniques that better fit the bill here — or even just want to talk about potential features you think are missing in the services discussed here — I’d love to hear from you via Twitter or via email .