LlamaIndex - Building a RAG pipeline
LlamaIndex - Building a RAG pipeline 1 Loading & Ingestion Load the data Transform the data Index and store the data 1.1 Loaders The way LlamaIndex does this is via data connectors, also called Reader 1 2 3 4 5 6 7 #读取文件夹： from llama_index.core import SimpleDirectoryReader documents = SimpleDirectoryReader("./data").load_data() #读取单个文档 from llama_index.core import Document doc = Document(text="text") 1.2 Transformations After the data is loaded, you then need to process and transform your data before putting it into a storage system. These transformations include chunking, extracting metadata, and embedding each chunk. This is necessary to make sure that the data can be retrieved, and used optimally by the LLM. ...