What is Discord Vector Db

discord_vector_db is a tool designed for retrieving messages from Discord channels and storing them in a vector database for semantic search and analysis.

Use cases

Use cases include analyzing community feedback, retrieving historical messages for research, ensuring compliance with privacy regulations, and enhancing customer support interactions through semantic search.

How to use

To use discord_vector_db, clone the repository, set up a virtual environment, install dependencies, and utilize the provided Python scripts to fetch messages and process them for the vector database.

Key features

Key features include robust message retrieval with pagination support, privacy protection through PII detection and redaction, resilient design with error recovery, vector database integration for semantic search, and built-in ethical considerations for privacy controls.

Where to use

discord_vector_db can be used in various fields such as data analysis, customer support, community management, and any application requiring semantic search capabilities on Discord messages.

Discord Message Vector DB

A tool for retrieving Discord messages and storing them in a vector database for semantic search and analysis.

Overview

This project provides a secure, privacy-respecting way to retrieve messages from Discord channels using the Discord MCP Server, process them for privacy concerns, and store them in a vector database (ChromaDB) for semantic search and analysis.

Features

• Robust Message Retrieval: Fetch all messages from Discord channels with pagination support
• Privacy Protection: PII detection and redaction, opt-out registry
• Resilient Design: Checkpointing, error recovery, and circuit breaker patterns
• Vector Database Integration: Convert messages to embeddings for semantic search
• Ethical Considerations: Built-in privacy controls and data minimization

Installation

# Clone the repository
git clone https://github.com/yourusername/discord_vector_db.git
cd discord_vector_db

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Usage

Basic Usage

from discord_retriever.fetcher import DiscordMessageFetcher

# Initialize fetcher
fetcher = DiscordMessageFetcher(
    channel_id="731607577481314359",
    save_directory="messages"
)

# Retrieve all messages
fetcher.fetch_all()

Processing for Vector Database

from discord_retriever.processor import process_for_vector_db

# Process messages and add to vector database
collection = process_for_vector_db(
    messages_directory="messages",
    collection_name="discord_messages"
)

# Search for semantically similar messages
results = collection.query(
    query_texts=["tell me about security concerns"],
    n_results=5
)

Command Line Interface

# Fetch messages from a Discord channel
python -m discord_retriever.cli fetch --channel-id 731607577481314359 --save-dir messages

# Process messages for vector database
python -m discord_retriever.cli process --messages-dir messages --collection discord_messages

# Search for messages
python -m discord_retriever.cli search --collection discord_messages --query "security concerns"

Ethical Considerations

This tool is designed with privacy and ethics in mind:

• All personal identifiable information (PII) can be automatically redacted
• Users can opt-out of having their messages processed
• Data minimization principles are applied by default
• Secure storage options for sensitive data

License

MIT

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details.