Skip to main content

RAG Parameter Settings

System Prompt

Enabled​

The system preserves the current system prompt, assistant prompt, and conversation history. This allows the model to understand the entire interaction more comprehensively and generate more accurate, context-aware responses.

Disabled​

The system ignores the system prompt and responds using only the assistant prompt. While this can be useful for certain specialized tasks where clean, context-independent responses are desired, it generally weakens the overall coherence of the conversation.

Merge Chunking

Merge Chunking is the process of combining text chunks that are semantically or topically related within Retrieval-Augmented Generation (RAG) systems. This creates larger, more coherent context blocks, allowing the model to receive more complete information and improve response quality.

Search Limit

Search Limit restricts the maximum number of results an agent can retrieve from a knowledge source (e.g., a vector database) for a given query. A higher value provides access to a broader information pool, helping generate more comprehensive responses.

Configuration for Full Coverage​

For example, if an uploaded document is split into 100 vectors, setting the Search Limit to 100 ensures that all chunks are considered during the initial retrieval stage. This eliminates the risk of missing critical lines or sections.

Note: The maximum available limit is displayed above the Search Limit slider after a knowledge base document is uploaded and the assistant is saved.

Top-K

In RAG systems, Top-K selects the K most relevant results from the larger retrieved result set.

Example​

If a search returns 100 results and Top-K is set to 5, the system will use only the 5 most relevant results.

Difference Between Search Limit and Top-K​

  • Search Limit determines how many results are initially retrieved.
  • Top-K determines how many of those retrieved results are ultimately selected.

They work together:

  • Search Limit: "Which pool of results should I retrieve?"
  • Top-K: "How many of these results should actually be passed to the LLM?"

In scenarios where no information loss is acceptable, both values can be set equally (e.g., 40–40).

Practical Considerations​

Higher values​

  • Increase recall by capturing deeper information.
  • Increase latency and cost.

Lower values​

  • Improve response speed.
  • Increase the risk of missing relevant content.

Search Threshold

Search Threshold defines how similar a piece of information must be to the user's query in order to be considered relevant. Results scoring below the threshold are ignored.

Technical Details and Best Practices​

For advanced users:

  • In vector search, similarity scores are typically normalized between 0 and 1.
  • Scores closer to 1 indicate stronger similarity.

Low Threshold​

  • Returns more results.
  • Increases the chance of including irrelevant content.

High Threshold​

  • Returns fewer but more relevant results.
  • Increases the risk of excluding useful information.

Practical Recommendation​

If reranking is enabled, the Search Threshold can be set lower. This allows a broader set of candidates to be retrieved initially, while less relevant results are filtered out during the reranking stage.

Reranking

Reranking is the process of re-scoring and reordering initial search results using a secondary model to ensure that the most relevant documents appear first.

Enabled​

An advanced model reranks the retrieved documents to improve relevance and accuracy.

Disabled​

The reranking step is skipped, resulting in faster responses but potentially lower retrieval quality.

Rerank Top-N

After reranking, the Top-N most relevant results are selected and passed to the LLM. This helps the model focus on the most useful information while avoiding unnecessary context overload.

Since LLMs have limited context windows (e.g., 4K, 8K, 32K tokens), sending too many documents can negatively impact performance.

Single, Precise-Answer Questions​

Keep N small (3–5).

Comparisons, Summaries, or Multi-Perspective Analyses​

Use a medium to high N value (10–20).

Practical Tips​

  • If your dataset is small and well-structured, reranking may not be necessary.
  • For large or heterogeneous collections, reranking should generally be enabled.
  • For critical accuracy domains (medical, legal): use a higher threshold and a lower Top-N.
  • For creative, exploratory, or summarization tasks: use a medium threshold and a higher Top-N.

Rerank Threshold

Rerank Threshold defines the minimum score a document must achieve after reranking in order to be passed to the LLM. Documents scoring below this threshold are discarded, ensuring that only highly relevant content is included in the final context.

Threshold Guidelines​

High Threshold (≥ 0.7)​

  • Only highly relevant documents are retained.
  • High precision, lower coverage.

Low Threshold (≈ 0.1–0.4)​

  • More documents are retained.
  • Higher coverage, but with an increased risk of including irrelevant content.