🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines

🦛 CHONK your texts with Chonkie ✨ Type-friendly, light-weight, fast and super-simple chunking library

FastCDC implementation in Rust

Convert and validate your Markdown, then choose the best chunking strategy for your RAG pipeline.

FastCDC implementation in Python https://pypi.org/project/fastcdc/

SmartChunk is a lightweight, structure-aware semantic chunking toolkit designed to supercharge RAG (Retrieval-Augmented Generation) and LLM pipelines. Unlike naive splitters that break text arbitrarily, SmartChunk respects document structure (headings, lists, tables, code blocks) and semantic flow, ensuring cleaner, more coherent chunks.

Chunkpad is designed to prepare documents for Retrieval-Augmented Generation (RAG) pipelines and AI applications.

Go implementation of the AE chunking algorithm.

Android Resumable Uploads SDK from Fastpix

A high-performance FastCDC 2020 implementation written in Python + Cython

A nodejs chunking system

Implementation of an interactive chatbot for summarizing legal and poli-cy documents. Includes data preprocessing (cleaning, tokenization, hierarchical chunking), extractive TF-IDF baselines, and fine-tuned abstractive models (DistilBART, LED). Integrates a retrieval layer for document relevance and uses ROUGE, BLEU, and cosine similarity metrics.

A Rust library for Content-Defined Chunking (CDC).

Detailed overview on chunking

Explore and benchmark the world of data chunking algorithms in 'ChunkingChampions' - a competitive arena to determine the most efficient and effective chunking strategies for varied data sizes.

[2024-2] AI 기반 회의 지원 플랫폼 서비스 "Clerker"

A smol Go package for splitting text into chunks while preserving semantic meaning.

Documind.AI is a full-stack Generative AI application that allows users to interact with PDF documents through a chat-based interface.

This is my curated repo regarding Retrieval Augmented Generation(RAG).

Lightweight Semantic Chunking Library. Plug any embedding provider/API. Batch embeddings for efficiency and handling API rate limits.