How We Built a Slack Bot for a UCF Research Park Startup

<i>An anonymized case study of a fast-growing startup in Orlando's UCF Research Park: we built a Slack bot grounded in their messy Notion wiki, stopped 30 recurring questions, and saved the ops team 12 hours a week.</i>

(Client details are anonymized and some specifics composited at the client’s request.)

It was a Tuesday afternoon in early 2024. I was sitting in a small conference room at a startup in the UCF Research Park — you know the type: exposed brick, a wall of whiteboards covered in half-erased diagrams, and the hum of a server closet that was way too warm. The founder, a former UCF engineering professor, had a problem that was eating his lunch.

His company had grown from 12 to 35 people in 18 months. Every week, a new hire would start, and every week, the same questions would flood the #general Slack channel. Things like:

• “How do I set up my development environment?”
• “What’s the PTO policy?”
• “Where’s the Wi-Fi password?”
• “How do I expense a new monitor?”
• “Who do I talk to about AWS permissions?”

He estimated that his senior engineers were fielding 30+ of these questions per new hire in the first two weeks. That’s 30 interruptions per person, per new hire. With three hires a month, that’s 90 interruptions — roughly 12 hours of lost deep work every month, just answering things that were already written down somewhere.

“We have a Notion wiki,” he said, pushing his glasses up. “But nobody reads it. And honestly, it’s a mess.”

I’d seen this pattern a dozen times. The wiki had grown organically — a page on office policies here, a dev setup guide there, some meeting notes, some outdated references. The search didn’t work. People defaulted to asking a human because it was faster than digging through the Notion. So we decided to build a Slack bot that could answer those questions instantly, pulling from their existing docs.

The Situation: What Was Breaking

Let me paint a clearer picture. This startup was in the defense-tech space — they built simulation software for military training. Their team was a mix of software engineers, former military SMEs, and business development folks. The churn rate for new hires was low, but the ramp time was brutal. It typically took 6–8 weeks before a new engineer felt productive.

The core problem wasn’t a lack of documentation. It’s that the documentation lived in a messy Notion workspace with 400+ pages, many of which were outdated or duplicated. When a new hire searched for “how to set up my dev environment,” they’d get three different pages, each with conflicting instructions. So they’d ask in Slack, and someone would answer from memory — often incorrectly.

The ops team (a single person, the founder’s wife) was spending 6 hours a week just answering onboarding questions. She had a folder of canned responses, but she still had to copy-paste them. The senior engineers, meanwhile, were getting pinged with questions like “What’s the SSH command for the staging server?” — which was literally on page 2 of the onboarding doc.

We needed a way to make the knowledge accessible without requiring people to read 400 pages. And we needed to do it without asking the team to change their behavior. They lived in Slack. The answer had to live in Slack too.

What They’d Tried Before

Before they called me, they’d tried a few things. The founder had purchased a license for a popular internal knowledge base tool that had a Slack integration. It allowed you to ask questions in Slack and it would search the knowledge base and return an answer. The problem was that it required you to manually curate and tag every page — and nobody had time to do that. After a month, it had been used exactly 3 times.

They also tried a simple FAQ channel in Slack, pinned with a document. But new hires didn’t scroll up. They just asked. And the pinned doc got stale.

They’d also considered building a simple Q&A bot using a spreadsheet and some regex. But the founder, being a former professor, wanted something “smarter” — something that could understand natural language and handle variations of the same question.

“I want it to work like ChatGPT,” he said. “But trained on our docs.”

I nodded. That’s exactly what we’d build — but with a few important guardrails.

The AI Work: What We Actually Built

We built a Slack bot using a Retrieval-Augmented Generation (RAG) architecture. Here’s the plain-English version of what that means: When a user asks a question in Slack, the bot searches through their Notion docs for the most relevant chunks of text, then passes those chunks to a large language model (LLM) along with the question. The LLM generates an answer based only on those chunks — not on its general training data. This keeps the answers grounded in the company’s actual docs and reduces hallucinations.

But here’s the catch. Their Notion workspace was messy — really messy. Pages had inconsistent formatting, nested content, images with no alt text, and lots of duplicate information. We couldn’t just dump the entire workspace into a vector database and call it done. We had to clean it up first.

Step 1: Data extraction and cleaning

We used the Notion API to pull all pages from their workspace. We wrote a Python script that traversed the page tree, extracted text content from each block, and flattened it into a single document per page. We stripped out images (we’d handle those seperately), removed empty blocks, and concatenated text from nested bullet lists.

Then we chunked each page into overlapping segments of about 500 tokens each, with a 50-token overlap. This is a standard technique to make sure the LLM doesn’t miss context that spans across chunk boundaries. We used LangChain’s text splitter for this.

Step 2: Generating embeddings

For each chunk, we generated a vector embedding using OpenAI’s text-embedding-ada-002 model. These embeddings are numerical representations of the text’s meaning. We stored them in a Pinecone vector database, which allows for fast similarity search. When a user asks a question, we embed the question the same way, then search Pinecone for the top 5 most similar chunks.

Step 3: Building the Slack bot

We built the bot using Slack’s Bolt framework in Python. The bot listens for direct messages and mentions in public channels. When it receives a message, it:

1. Embeds the message text
2. Searches Pinecone for the top 5 relevant chunks
3. Passes those chunks as context to GPT-4, along with a system prompt that says: “You are an onboarding assistant for [Company]. Answer the user’s question based only on the provided context. If the context doesn’t contain the answer, say ‘I don’t know. Let me find someone who can help.’ Be concise.”
4. Returns the generated answer to Slack

We also added a feedback mechanism: thumbs up / thumbs down reactions on each answer, which we logged to a database. This let us monitor which answers were working and which needed improvement.

Step 4: Hardcoding the 30 most common questions

During the first week of testing, we noticed something: about 30 questions made up 80% of the requests. Things like “Wi-Fi password,” “PTO request process,” “Who is the IT contact?” These were simple, factual answers that didn’t really need the RAG pipeline. So we created a lookup table of these 30 questions and their canonical answers, curated with the ops team. The bot checks this table first. If it finds a match (using fuzzy matching via thefuzz library), it returns the hardcoded answer instantly. This cut response time from ~3 seconds to under 500ms for those questions.

Step 5: Keeping a human in the loop

We deliberately didn’t make the bot fully autonomous. For any question it couldn’t answer (i.e., it returned “I don’t know”), we had it post a message to a private Slack channel called #onboarding-bot-escapes. The ops team would see it and either answer directly or update the Notion docs. Then we’d re-index that page so the bot would know the answer next time. This human-in-the-loop loop was critical for catching gaps and improving the knowledge base over time.

We also set up a weekly review where the ops team would go through the feedback logs and identify which answers had negative reactions. They’d then update the Notion docs accordingly, and we’d re-run the indexing pipeline. Over three months, the bot’s accuracy went from about 70% to over 95%.

Where It Was Harder Than Expected

I’ll be honest: the Notion data was messier than I anticipated. We spent a full week just cleaning and deduplicating. There were pages with the same title but different content, pages that were just a single image, and pages that were entirely empty except for a title. We had to write custom logic to handle these edge cases.

Another challenge was handling questions that required multi-step reasoning. For example, a new hire asked: “I need to set up my dev environment, but I’m on a Mac and the guide says Windows.” The bot initially returned the Windows guide because it was the most relevant chunk. We had to improve the retrieval by adding metadata tags (e.g., OS, department) and filtering based on the user’s profile. We solved this by having the bot ask clarifying questions when the context was ambiguous, using a simple chain-of-thought prompt.

Also, we underestimated how much response formatting mattered. The first version returned plain text with no formatting. Users complained that it was hard to read multi-step instructions. We switched to returning Slack’s mrkdwn format, with bullet points and bold headers. That made a big difference.

The Measured Results

After three months of operation, here’s what we saw:

• 12 hours/week saved for the ops team on onboarding questions. That’s 48 hours a month — essentially an extra week of work.
• 30% reduction in Slack pings to senior engineers about onboarding topics. They reported feeling less interrupted.
• 3 weeks cut from the average new hire ramp time. New engineers reported feeling productive after 3–4 weeks instead of 6–8.
• 95% answer accuracy (based on user feedback) for questions the bot attempted to answer. The bot handled about 60% of all onboarding questions autonomously; the remaining 40% were either hardcoded (30%) or escalated to humans (10%).

One specific metric that surprised me: the number of questions per new hire dropped from 30 to about 8 in the first two weeks. That’s a 73% reduction. The founder told me that the bot “paid for itself in the first month” — though I’ll note that the actual cost was about $200/month in OpenAI API fees plus the Pinecone and hosting costs.

What We’d Do Differently

If I had to do this again, there are a few things I’d change.

First, I’d start with a readiness assessment of their existing documentation before writing any code. We spent alot of time cleaning data that could have been avoided with better up-front organization. I’d also recommend they do a one-time cleanup of their Notion workspace before we even start the project. It would’ve saved us a week.

Second, I’d invest more in the feedback loop. The thumbs up/down was helpful, but we missed a lot of nuance. We should’ve added a way for users to suggest edits directly through the bot — like “This answer is wrong because…” — and have that create a Notion task automatically.

Third, I’d consider using a more specialized embedding model for technical content. The generic ada-002 model worked fine, but a model fine-tuned on code or engineering docs might’ve handled the dev environment questions better.

Finally, I’d think more carefully about when to use RAG vs. fine-tuning. For this use case, RAG was right because the knowledge base changes frequently. But for a more static FAQ, fine-tuning a smaller model might be cheaper and faster. We actually explored fine-tuning a GPT-3.5 model on their Q&A pairs, but the data wasn’t large enough to justify it. Maybe in the future.

Closing Thoughts

This project was a classic example of what I call “knowledge accessibility” — making the information you already have actually findable and usable. The bot didn’t create new knowledge; it just made the existing knowledge easier to reach. And by doing that, it saved real time and money for a growing Central Florida startup.

If you’re a business owner in Orlando — whether you’re in Lake Nona, Winter Park, or right here in the Research Park — and you’re watching your team drown in repetitive questions, this approach might work for you. It’s not magic. It’s just good engineering with a human touch.

Want to see if your own knowledge base is ready for a bot like this? We offer a free AI readiness assessment that looks at your documentation quality, team size, and pain points. And if you’re ready to build, we can help you scope an AI voice agent or Slack bot that fits your budget.

Or, if you’re just starting to think about AI for your business, check out our AI glossary to understand the terms. And when you’re ready to talk, reach out. We’d love to help.

"The bot didn't create new knowledge; it just made the existing knowledge easier to reach."

Frequently asked questions

How much did the Slack bot cost to build?

The initial build took about 3 weeks of part-time work, roughly $5,000 in consulting fees. Ongoing costs are about $200/month for OpenAI API, Pinecone vector database, and hosting. The bot saved the ops team 12 hours/week, which more than covered the cost in the first month.

Did the bot completely replace human support?

No. We deliberately kept a human in the loop. The bot handles about 60% of questions autonomously, another 30% are answered from a hardcoded lookup table, and the remaining 10% are escalated to a human via a private Slack channel. This ensures accuracy and continuous improvement.

What if the Notion docs are really messy?

We spent a week cleaning and deduplicating the client's Notion workspace. If your docs are extremely messy, we can help you organize them first as part of a readiness assessment. The cleaner the data, the better the bot performs.

Can this work for a company with fewer than 10 employees?

Probably not worth the investment. The bot starts paying for itself when you have at least 15–20 employees and regular new hires. For smaller teams, a well-organized FAQ document and a pinned Slack message might be enough.

What if the bot gives a wrong answer?

Users can give a thumbs-down reaction, which logs the answer for review. The ops team reviews these weekly and updates the Notion docs if needed. We also have the bot say 'I don't know' when it's not confident, rather than guessing.

Do we need to use Notion? What about Confluence or Google Docs?

We can build the bot with any source that has an API: Notion, Confluence, Google Docs, SharePoint, etc. The RAG approach is source-agnostic. We just need to extract the text and generate embeddings.