init. project
This commit is contained in:
24
.github/AGENTS.md
vendored
Normal file
24
.github/AGENTS.md
vendored
Normal file
@@ -0,0 +1,24 @@
|
||||
# 测试生成智能体约定
|
||||
|
||||
## 适用范围
|
||||
- 本工作区包含基于 Tool Calling 与 Skill Calling 的测试内容生成链路。
|
||||
- 当用户提出测试项分解、测试用例生成或预期成果生成需求时，必须触发 testing-orchestrator。
|
||||
|
||||
## 已注册技能
|
||||
- identify-requirement-type：将用户需求文本识别为明确的测试需求类型，为后续测试项分解与测试用例生成提供分类依据。
|
||||
- decompose-test-items：按需求类型规则生成正常测试与异常测试测试项。
|
||||
- generate-test-cases：按测试项生成可执行测试用例，至少 1 条/测试项。
|
||||
- testing-orchestrator：按标准顺序编排工具调用并输出结构化结果。
|
||||
|
||||
## 强制调用链
|
||||
1. identify-requirement-type
|
||||
2. decompose-test-items
|
||||
3. generate-test-cases
|
||||
4. build_expected_results
|
||||
5. format_output
|
||||
|
||||
## 约束规则
|
||||
- 除非用户明确要求只看中间步骤，否则禁止跳步。
|
||||
- 每一步都必须显式接收上一步输出作为上下文输入。
|
||||
- 若无法识别类型，必须输出未知类型及候选类型，并继续执行通用分解。
|
||||
- 最终输出必须严格包含测试项、测试用例、预期成果三段，并按正常测试/异常测试分组。
|
||||
46
.github/skills/decompose-test-items/SKILL.md
vendored
Normal file
46
.github/skills/decompose-test-items/SKILL.md
vendored
Normal file
@@ -0,0 +1,46 @@
|
||||
---
|
||||
name: decompose-test-items
|
||||
description: "当需要基于需求类型把需求文本分解为可执行的正常/异常测试项时使用。"
|
||||
---
|
||||
|
||||
# decompose-test-items
|
||||
|
||||
## 目标
|
||||
基于用户需求文本和已识别需求类型，生成测试项列表。
|
||||
|
||||
## 输入
|
||||
- user_requirement_text
|
||||
- requirement_type
|
||||
|
||||
## 输出
|
||||
- normal_test_items：完整、可执行的正常测试项列表。
|
||||
- abnormal_test_items：完整、可执行的异常测试项列表。
|
||||
|
||||
## 强制规则
|
||||
1. 每个软件功能至少应被正常测试与被认可的异常场景覆盖；复杂功能需继续细分。
|
||||
2. 每个测试项必须语义完整、可直接执行。
|
||||
3. 覆盖必须包含：正常流程、边界条件（适用时）、异常条件。
|
||||
4. 粒度需适中，避免过粗或过细。
|
||||
5. 对未知类型必须执行通用分解，并保持正常/异常分组。
|
||||
6. 对需求说明未显式给出但在用户手册或操作手册体现的功能，也应补充测试项覆盖。
|
||||
|
||||
## 14类最小分解检查点
|
||||
- 功能测试：正常覆盖功能主路径、基本数据类型、合法边界值与状态转换；异常覆盖非法输入、不规则输入、非法边界值与最坏情况。
|
||||
- 性能测试：正常覆盖处理精度、响应时间、处理数据量与模块协调性；异常覆盖超负荷、软硬件限制、负载潜力上限与资源占用异常。
|
||||
- 外部接口测试：正常覆盖全部外部接口格式与内容正确性；异常覆盖每个输入输出接口的错误格式、错误内容与异常交互。
|
||||
- 人机交互界面测试：正常覆盖界面风格一致性与标准操作流程；异常覆盖误操作、快速操作、非法输入、错误命令与错误流程提示。
|
||||
- 强度测试：正常覆盖设计极限下系统功能和性能表现；异常覆盖超出极限时的降级行为、健壮性与饱和表现。
|
||||
- 余量测试：正常覆盖存储、通道、处理时间余量是否满足要求；异常覆盖余量不足或耗尽时系统告警与受控行为。
|
||||
- 可靠性测试：正常覆盖典型环境、运行剖面与输入变量组合；异常覆盖失效等级场景、边界环境变化、不合法输入域及失效记录。
|
||||
- 安全性测试：正常覆盖安全关键部件、安全结构与合法操作路径；异常覆盖危险状态、故障模式、边界接合部、非法进入与数据完整性保护。
|
||||
- 恢复性测试：正常覆盖故障探测、备用切换、恢复后继续执行；异常覆盖故障中作业保护、状态保护与恢复失败路径。
|
||||
- 边界测试：正常覆盖输入输出域边界、状态转换端点与功能界限；异常覆盖性能界限、容量界限和越界端点。
|
||||
- 安装性测试：正常覆盖标准及不同配置下安装卸载流程；异常覆盖安装规程错误、依赖异常与中断后的处理。
|
||||
- 互操作性测试：正常覆盖两个或多个软件同时运行与互操作过程；异常覆盖互操作失败、并行冲突与协同异常。
|
||||
- 敏感性测试：正常覆盖有效输入类中典型数据组合；异常覆盖引发不稳定或不正常处理的特殊数据组合。
|
||||
- 测试充分性要求：正常覆盖需求覆盖率、配置项覆盖与代码覆盖达标；异常覆盖未覆盖部分逐项分析、确认与报告输出。
|
||||
|
||||
## 未知类型容错
|
||||
- 当 requirement_type 无法确定时，仍需输出正常/异常两组测试项。
|
||||
- 通用正常项至少包含：主流程正确性、合法边界值、标准输入输出。
|
||||
- 通用异常项至少包含：非法输入、越界输入、资源异常或状态冲突。
|
||||
45
.github/skills/generate-test-cases/SKILL.md
vendored
Normal file
45
.github/skills/generate-test-cases/SKILL.md
vendored
Normal file
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: generate-test-cases
|
||||
description: "当需要根据已分解测试项生成包含操作步骤与测试内容的具体测试用例时使用。"
|
||||
---
|
||||
|
||||
# generate-test-cases
|
||||
|
||||
## 目标
|
||||
按测试项生成测试用例，每个测试项至少对应 1 条用例。
|
||||
|
||||
## 输入
|
||||
- normal_test_items
|
||||
- abnormal_test_items
|
||||
|
||||
## 输出
|
||||
- normal_test_cases
|
||||
- abnormal_test_cases
|
||||
|
||||
每条测试用例必须包含：
|
||||
- operation_steps
|
||||
- test_content
|
||||
- expected_result_placeholder
|
||||
|
||||
## 规则
|
||||
1. 测试项与测试用例应保持一一对应关系。
|
||||
2. 每个测试项必须至少生成 1 条测试用例。
|
||||
3. 必须区分正常测试用例与异常测试用例。
|
||||
4. 操作步骤应可顺序执行，避免歧义。
|
||||
5. 操作步骤必须包含明确动作、对象和输入条件，禁止笼统动作词。
|
||||
6. test_content 必须包含可验证条件，便于后续生成可度量预期成果。
|
||||
|
||||
## expected_result_placeholder 映射
|
||||
- {{return_value}}：接口或函数返回值验证。
|
||||
- {{state_change}}：系统状态变化验证。
|
||||
- {{error_message}}：异常场景错误信息验证。
|
||||
- {{data_persistence}}：数据库或存储落库结果验证。
|
||||
- {{ui_display}}：界面显示反馈验证。
|
||||
|
||||
## 禁止模糊描述
|
||||
- 错误示例："检查功能正常"；正确示例："验证返回状态码为200且响应体包含status=success"。
|
||||
- 错误示例："输入合法数据"；正确示例："在用户名输入框输入长度为8的字母数字字符串并提交"。
|
||||
- 错误示例："系统提示错误"；正确示例："触发非法输入后显示错误码E400和字段级提示文案"。
|
||||
|
||||
## 预期结果耦合
|
||||
- 每条用例必须可在下一步绑定一条明确、可验证的预期成果。
|
||||
62
.github/skills/identify-requirement-type/SKILL.md
vendored
Normal file
62
.github/skills/identify-requirement-type/SKILL.md
vendored
Normal file
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: identify-requirement-type
|
||||
description: "当需要在测试项分解与测试用例生成之前识别需求类型时使用。"
|
||||
---
|
||||
|
||||
# identify-requirement-type
|
||||
|
||||
## 目标
|
||||
将用户需求文本识别为明确的测试需求类型，为后续测试项分解与测试用例生成提供分类依据。
|
||||
|
||||
## 输入
|
||||
- user_requirement_text：用户原始需求文本。
|
||||
|
||||
## 输出
|
||||
- requirement_type：以下之一
|
||||
- 功能测试
|
||||
- 性能测试
|
||||
- 外部接口测试
|
||||
- 人机交互界面测试
|
||||
- 强度测试
|
||||
- 余量测试
|
||||
- 可靠性测试
|
||||
- 安全性测试
|
||||
- 恢复性测试
|
||||
- 边界测试
|
||||
- 安装性测试
|
||||
- 互操作性测试
|
||||
- 敏感性测试
|
||||
- 测试充分性要求
|
||||
- 未知类型
|
||||
- reason：简要判断依据。
|
||||
- candidates：当 requirement_type 为未知类型时，给出 1-3 个最接近候选类型。
|
||||
|
||||
## 类型识别信号
|
||||
- 功能测试：关注功能需求逐项验证、业务流程正确性、输入输出行为、状态转换与边界值处理。
|
||||
- 性能测试：关注处理精度、响应时间、处理数据量、系统协调性、负载潜力与运行占用空间。
|
||||
- 外部接口测试：关注外部输入输出接口的格式、内容、协议与正常/异常交互表现。
|
||||
- 人机交互界面测试：关注界面一致性、界面风格、操作流程、误操作健壮性与错误提示能力。
|
||||
- 强度测试：关注系统在极限、超负荷、饱和和降级条件下的稳定性与承受能力。
|
||||
- 余量测试：关注存储余量、输入输出通道余量、功能处理时间余量等资源裕度。
|
||||
- 可靠性测试：关注真实或仿真环境下的失效等级、运行剖面、输入覆盖和长期稳定运行能力。
|
||||
- 安全性测试：关注危险状态响应、安全关键部件、异常输入防护、非法访问阻断和数据完整性保护。
|
||||
- 恢复性测试：关注故障探测、备用切换、系统状态保护与从无错误状态继续执行能力。
|
||||
- 边界测试：关注输入输出域边界、状态转换端点、功能界限、性能界限与容量界限。
|
||||
- 安装性测试：关注不同配置下安装卸载流程和安装规程执行正确性。
|
||||
- 互操作性测试：关注多个软件并行运行时的互操作能力与协同正确性。
|
||||
- 敏感性测试：关注有效输入类中可能引发不稳定或不正常处理的数据组合。
|
||||
- 测试充分性要求：关注需求覆盖率、配置项覆盖、语句覆盖、分支覆盖及未覆盖分析确认。
|
||||
|
||||
## 规则
|
||||
1. 优先依据需求文本中的显式表述进行分类。
|
||||
2. 分类应以语义意图为主，不能只做关键词机械匹配。
|
||||
3. 置信度不足时输出未知类型，并提供候选类型。
|
||||
4. 判断依据需简洁、可追溯到文本证据。
|
||||
|
||||
## 容错
|
||||
- 当需求描述过于笼统或跨多类型混合时，输出未知类型，并在 candidates 给出最接近类型。
|
||||
- 当识别不稳定时，优先保守分类，不强行归入单一类型。
|
||||
- 未知类型不阻断后续流程，应继续执行通用测试项分解。
|
||||
|
||||
## 调试
|
||||
- debug 模式下返回每个类型的分类分数 classification_scores。
|
||||
57
.github/skills/testing-orchestrator/SKILL.md
vendored
Normal file
57
.github/skills/testing-orchestrator/SKILL.md
vendored
Normal file
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: testing-orchestrator
|
||||
description: "当用户要求测试项分解或测试用例生成且需要完整工具调用链时使用。"
|
||||
---
|
||||
|
||||
# testing-orchestrator
|
||||
|
||||
## 目标
|
||||
严格执行测试生成调用链，并显式传递每一步上下文。
|
||||
|
||||
## 标准调用链
|
||||
1. identify-requirement-type
|
||||
2. decompose-test-items
|
||||
3. generate-test-cases
|
||||
4. build_expected_results
|
||||
5. format_output
|
||||
|
||||
## 编排规则
|
||||
1. 优先使用 Skill 与 Tool，不使用临时硬编码逻辑替代。
|
||||
2. 除非用户明确要求，否则不得跳过任何步骤。
|
||||
3. 每一步必须显式接收上一步输出。
|
||||
4. 分类失败时输出未知类型并继续执行通用分解。
|
||||
|
||||
## 输出模板
|
||||
最终输出必须严格遵循以下分组结构：
|
||||
|
||||
**测试项**
|
||||
|
||||
**正常测试**：
|
||||
1. [测试项 N1]：...
|
||||
|
||||
**异常测试**：
|
||||
1. [测试项 E1]：...
|
||||
|
||||
**测试用例**
|
||||
|
||||
**正常测试**：
|
||||
1. [用例 N1]（对应测试项 N1）：...
|
||||
|
||||
**异常测试**：
|
||||
1. [用例 E1]（对应测试项 E1）：...
|
||||
|
||||
**预期成果**
|
||||
|
||||
**正常测试**：
|
||||
1. [预期 N1]（对应用例 N1）：...
|
||||
|
||||
**异常测试**：
|
||||
1. [预期 E1]（对应用例 E1）：...
|
||||
|
||||
## 调试模式
|
||||
当 debug=true 时，输出步骤日志并包含：
|
||||
- step_name
|
||||
- input_summary
|
||||
- output_summary
|
||||
- success
|
||||
- fallback_used
|
||||
108
.github/测试项分解要求.md
vendored
Normal file
108
.github/测试项分解要求.md
vendored
Normal file
@@ -0,0 +1,108 @@
|
||||
5.4.5.1功能测试
|
||||
功能测试是对软件需求规格说明中的功能需求逐项进行的测试，以验证其功能是否满足要求。功能测试一般需进行：
|
||||
1. 每一个软件功能应至少被一个测试用例和一个被认可的异常所覆盖，对大的功能应进一步分解为更细的功能，使测试用例可以直接和功能对应：
|
||||
2. 用基本数据类型和数据值测试：
|
||||
3. 用一系列合理的数据类型和数据值运行，测试超负荷、饱和及其它“最坏情况”的结果；
|
||||
4. 用假想的数据类型和数据值运行，测试排斥不规则输入的能力；
|
||||
5. 每个功能的合法边界值和非法边界值都应被作为测试用例；
|
||||
6. 应考虑软件功能对操作模式、运行环境、运行状态、状态转换、运行时间等的覆盖要求；
|
||||
7. 对于在需求规格说明中没有指明，而在用户使用手册、操作手册中表明出来的每一功能及操作，都应有相应测试用例覆盖。
|
||||
|
||||
5.4.5.2性能测试
|
||||
性能测试是对软件需求规格说明中的性能需求逐项进行的测试，以验证其性能是否满足要求。性能测试一般需进行：
|
||||
1. 测试程序在获得定量结果时程序计算的精确性(处理精度. ;
|
||||
2. 测试程序在有速度要求时完成功能的时间(响应时间. ;
|
||||
3. 测试程序完成功能所能处理的数据量；
|
||||
4. 测试程序各部分的协调性，如高速、低速操作的协调：
|
||||
5. 测试软/硬件中因素是否限制了程序的性能；
|
||||
6. 测试程序的负载潜力；
|
||||
7. 测试程序运行占用的空间。
|
||||
|
||||
5.4.5.3外部接口测试
|
||||
外部接口测试是对软件需求规格说明中的外部接口需求逐项进行的测试。外部接口测试一般需进行：
|
||||
1. 测试所有外部接口，检查接口信息的格式及内容；
|
||||
2. 对每一个外部的输入/输出接口做正常和异常情况的测试。
|
||||
|
||||
5.4.5.4人机交互界面测试
|
||||
人机交互界面测试是对所有人机交互界面提供的操作和显示界面进行的测试，以检验是否满足用户的要求。人机交互界面测试一般需进行：
|
||||
1. 测试操作和显示界面及界面风格与软件需求规格说明中要求的一致性和符合性：
|
||||
2. 以非常规操作、误操作、快速操作来检验界面的健壮性；
|
||||
3. 测试对错误命令或非法数据输入的检测能力与提示情况；
|
||||
4. 测试对错误操作流程的检测与提示：
|
||||
5. 如果有用户手册或操作手册，应对照手册逐条进行操作和观察。
|
||||
|
||||
5.4.5.5强度测试
|
||||
强度测试是强制软件运行在不正常到发生故障的情况下(设计的极限状态到超出极限. ,检验软件可以运行到何种程度的测试。强度测试一般需进行：
|
||||
1. 性能的强度测试；
|
||||
2. 降级能力的强度测试；
|
||||
3. 系统健壮性测试；
|
||||
4. 系统饱和测试。
|
||||
强度测试在某种程度上可看作性能测试的延伸，测出软件功能、性能的实际极限。其详细要求可参见附录B“强度测试”。
|
||||
|
||||
5.4.5.6余量测试
|
||||
余量测试是对软件是否达到需求规格说明中要求的余量的测试。若无明确要求时，一般至少留有20%的余量。根据测试要求，余量测试一般需提供：
|
||||
1. 全部存储量的余量；
|
||||
2. 输入、输出及通道的余量；
|
||||
3. 功能处理时间的余量。
|
||||
|
||||
5.4.5.7可靠性测试
|
||||
可靠性测试是在真实的和仿真的环境中，为做出软件可靠性估计而对软件进行的功能测试(其输入覆盖和环境覆盖一般大于普通的功能测试. ,可靠性测试中必须按照运行剖面和使用的概率分布随机地选择测试用例。可靠性测试一般需：
|
||||
1. 测试环境应与典型使用环境的统计特性相一致，必要时使用测试平台；
|
||||
2. 定义软件失效等级；
|
||||
3. 建立软件运行剖面/操作剖面；
|
||||
4. 测试记录更为详细、准确，应记录失效现象和时间；
|
||||
5. 必须保证输入覆盖，应覆盖重要的输入变量值、各种使用功能、相关输入变量可能组合以及不合法输入域等；
|
||||
6. 对于可能导致软件运行方式改变的一些边界条件和环境条件，必须进行针对性测试。
|
||||
有关可靠性测试的详细要求参见附录C“可靠性测试”。
|
||||
|
||||
5.4.5.8安全性测试
|
||||
A、B、C级软件需要进行安全性测试。安全性测试是检验软件中已存在的安全性、安
|
||||
全保密性措施是否有效的测试。安全性测试一般：
|
||||
1. 应进行软件安全性分析，并且在软件需求中明确每一个危险状态及导致危险的可能原因，在测试中全面检验软件在这些危险状态下的反应；
|
||||
2. 对安全性关键的软件部件，应单独测试，以确认该软件部件满足安全性需求：
|
||||
3. 对软件设计中用于提高安全性的结构、算法、容错、冗余、中断处理等方案应进行针对性测试；
|
||||
4. 测试应尽可能在符合实际使用的条件下进行；
|
||||
5. 除在正常条件下测试外，应在异常条件下测试软件，以表明不会因可能的单个或多个输入错误而导致不安全状态；
|
||||
6. 应包含硬件及软件输入故障模式测试：
|
||||
7. 应包含边界、界外及边界接合部的测试；
|
||||
8. 应包括“0”、穿越“0”以及从两个方向趋近于“0”的输入值；
|
||||
9. 应包含在最坏情况配置下的最小和最大输入数据率，以确定系统的固有能力及对这些环境的反应；
|
||||
10. 操作员接口测试应包括在安全性关键操作中的操作员错误，以验证安全系统对这些错误的响应；
|
||||
11. 应测试双工切换、多机替换的正确性和连续性；
|
||||
12. 应测试防止非法进入系统并保护系统数据完整性的能力。
|
||||
|
||||
5.4.5.9恢复性测试
|
||||
恢复性测试是对有恢复或重置(reset. 功能的软件的每一类导致恢复或重置的情况，逐一进行的测试，以验证其恢复或重置功能。恢复性测试是要证实在克服硬件故障后，系统能否正常地继续进行工作，且不对系统造成任何损害。恢复性测试一般需进行：
|
||||
1. 探测错误功能的测试；
|
||||
2. 能否切换或自动启动备用硬件的测试；
|
||||
3. 在故障发生时能否保护正在运行的作业和系统状态的测试；
|
||||
4. 在系统恢复后，能否从最后记录下来的无错误状态开始继续执行作业的测试。
|
||||
|
||||
5.4.5.10边界测试
|
||||
边界测试是对软件处在边界或端点情况下运行状态的测试。边界测试一般需进行：
|
||||
1. 软件的输入域或输出域的边界或端点的测试；
|
||||
2. 状态转换的边界或端点的测试；
|
||||
3. 功能界限的边界或端点的测试；
|
||||
4. 性能界限的边界或端点的测试；
|
||||
5. 容量界限的边界或端点的测试。
|
||||
|
||||
5.4.5.11安装性测试
|
||||
安装性测试是对安装过程是否符合安装规程的测试，以发现安装过程中的错误。安装性测试一般需进行：
|
||||
1. 不同配置下的安装和卸载测试；
|
||||
2. 安装规程的正确性的测试。
|
||||
|
||||
5.4.5.12互操作性测试
|
||||
互操作性测试是为验证不同软件之间的互操作能力而进行的测试。互操作性测试一般：
|
||||
1. 必须同时运行两个或多个不同的软件；
|
||||
2. 软件之间发生互操作。
|
||||
|
||||
5.4.5.13敏感性测试
|
||||
敏感性测试是为发现在有效输入类中可能引起某种不稳定性或不正常处理的某些数据的组合而进行的测试。敏感性测试一般需进行：
|
||||
1. 发现有效输入类中可能引起某种不稳定性的数据组合的测试；
|
||||
2. 发现有效输入类中可能引起某种不正常处理的数据组合的测试。
|
||||
|
||||
5.4.5.14测试充分性要求
|
||||
1. 对软件需求规格说明中明确和隐含的需求(包括功能、性能、接口、质量要求等. 的覆盖率应达到100%;
|
||||
2. 配置项测试应使用与软件开发相同的编译器，全面覆盖软件需求说明文档中的所有要求。
|
||||
3. 对于A、B级嵌入式软件，对配置项源程序测试的语句、分支覆盖率均应达到100%。对用高级语言编制的A、B级嵌入式软件，应对配置项目标码进行结构分析和测试，测试的目标码语句、分支覆盖率均应达到100%。对覆盖率达不到要求的软件，应对未覆盖的部分逐一进行分析和确认，并提供分析报告。
|
||||
|
||||
164
README.md
Normal file
164
README.md
Normal file
@@ -0,0 +1,164 @@
|
||||
<div align="center">
|
||||
<img src="./docs/images/github-cover-new.png" alt="RAG Web UI" />
|
||||
<br />
|
||||
<p>
|
||||
<strong>基于 RAG 的知识库问答与文档处理平台</strong>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/rag-web-ui/rag-web-ui/blob/main/LICENSE"><img src="https://img.shields.io/github/license/rag-web-ui/rag-web-ui" alt="License" /></a>
|
||||
<a href="#"><img src="https://img.shields.io/badge/python-3.9+-blue.svg" alt="Python" /></a>
|
||||
<a href="#"><img src="https://img.shields.io/badge/node-%3E%3D18-green.svg" alt="Node" /></a>
|
||||
<a href="#"><img src="https://github.com/rag-web-ui/rag-web-ui/actions/workflows/test.yml/badge.svg" alt="CI" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="#简介">简介</a> •
|
||||
<a href="#核心能力">核心能力</a> •
|
||||
<a href="#项目结构">项目结构</a> •
|
||||
<a href="#快速开始">快速开始</a> •
|
||||
<a href="#配置说明">配置说明</a> •
|
||||
<a href="#api-概览">API 概览</a> •
|
||||
<a href="#常见问题">常见问题</a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
## 简介
|
||||
RAG Web UI 是一个面向企业知识库场景的全栈项目，提供从文档接入、向量化检索、对话问答到文档工程化处理的完整链路。
|
||||
|
||||
项目采用前后端分离架构：
|
||||
- 后端基于 FastAPI，负责认证、知识库、向量检索、文档处理任务与工具调用。
|
||||
- 前端基于 Next.js，提供知识库管理、聊天、文档处理等可视化页面。
|
||||
- 基础设施使用 MySQL、ChromaDB、MinIO，可通过 Docker Compose 一键启动。
|
||||
|
||||
## 核心能力
|
||||
- 知识库管理：支持 PDF、DOCX、Markdown、TXT 上传，预览、异步处理、状态追踪。
|
||||
- RAG 对话：支持多轮上下文问答，结合知识库检索结果生成回答。
|
||||
- 文档处理中心：提供需求提取、测试内容生成等工程化能力。
|
||||
- 工具中心：后端内置工具注册与调用机制，支持后续扩展更多可被模型调用的工具。
|
||||
- 多模型支持：支持 OpenAI、DashScope、DeepSeek、Ollama 等配置方式。
|
||||
|
||||
## 项目结构
|
||||
```text
|
||||
rag-web-ui/
|
||||
backend/ # FastAPI 后端
|
||||
app/
|
||||
api/api_v1/ # 业务 API 路由（auth/chat/knowledge-base/testing/tools）
|
||||
services/ # 文档处理、检索、向量存储等服务层
|
||||
tools/ # 工具中心（包含 SRS 需求提取工具）
|
||||
models/ # SQLAlchemy 模型
|
||||
schemas/ # Pydantic 数据模型
|
||||
alembic/ # 数据库迁移
|
||||
frontend/ # Next.js 前端
|
||||
src/app/ # 页面与路由
|
||||
src/components/ # 组件
|
||||
src/lib/ # API 封装、工具调用封装
|
||||
docs/ # 文档与教程
|
||||
docker-compose.yml # 标准部署编排
|
||||
docker-compose.dev.yml # 开发调试编排
|
||||
```
|
||||
|
||||
## 快速开始
|
||||
### 环境要求
|
||||
- Docker 与 Docker Compose v2+
|
||||
- Node.js 18+
|
||||
- Python 3.9+
|
||||
- 推荐 8GB 及以上内存
|
||||
|
||||
### 1. 准备环境变量
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
### 2. 启动服务
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
### 3. 访问服务
|
||||
- 前端页面: http://127.0.0.1.nip.io
|
||||
- API 文档: http://127.0.0.1.nip.io/redoc
|
||||
- MinIO 控制台: http://127.0.0.1.nip.io:9001
|
||||
|
||||
## 配置说明
|
||||
### 核心配置
|
||||
| 配置项 | 说明 | 示例 |
|
||||
| --- | --- | --- |
|
||||
| MYSQL_SERVER | MySQL 主机 | localhost 或 db |
|
||||
| MYSQL_PORT | MySQL 端口 | 3306 |
|
||||
| MYSQL_USER | MySQL 用户名 | ragagent |
|
||||
| MYSQL_PASSWORD | MySQL 密码 | ragagent |
|
||||
| MYSQL_DATABASE | MySQL 库名 | ragagent |
|
||||
| SECRET_KEY | JWT 密钥 | 自定义随机字符串 |
|
||||
| ACCESS_TOKEN_EXPIRE_MINUTES | Token 过期时间（分钟） | 10080 |
|
||||
|
||||
### 模型与向量配置
|
||||
| 配置项 | 说明 | 示例 |
|
||||
| --- | --- | --- |
|
||||
| CHAT_PROVIDER | 对话模型提供商 | dashscope/openai/deepseek/ollama |
|
||||
| EMBEDDINGS_PROVIDER | 向量模型提供商 | dashscope/openai/ollama |
|
||||
| DASH_SCOPE_API_KEY | DashScope Key | sk-xxx |
|
||||
| DASH_SCOPE_CHAT_MODEL | DashScope 对话模型 | qwen3.5-plus|
|
||||
| DASH_SCOPE_EMBEDDINGS_MODEL | DashScope 向量模型 | text-embedding-v4 |
|
||||
| VECTOR_STORE_TYPE | 向量库类型 | chroma 或 qdrant |
|
||||
|
||||
说明：
|
||||
- 当使用 DashScope 兼容模式时，建议使用文本向量模型（例如 text-embedding-v4）。
|
||||
- 项目支持通过 API_KEY 作为统一兜底密钥，再由各 provider 配置覆盖。
|
||||
|
||||
### 存储配置
|
||||
| 配置项 | 说明 | 示例 |
|
||||
| --- | --- | --- |
|
||||
| MINIO_ENDPOINT | MinIO 地址 | localhost:9000 |
|
||||
| MINIO_ACCESS_KEY | MinIO 用户名 | minioadmin |
|
||||
| MINIO_SECRET_KEY | MinIO 密码 | minioadmin |
|
||||
| MINIO_BUCKET_NAME | 桶名 | documents |
|
||||
|
||||
## API 概览
|
||||
后端统一前缀为 /api。
|
||||
|
||||
主要路由：
|
||||
- /api/auth: 登录、注册、令牌。
|
||||
- /api/knowledge-base: 知识库与文档上传/处理/任务状态。
|
||||
- /api/chat: 会话与消息。
|
||||
- /api/testing: 测试内容生成流水线。
|
||||
- /api/tools: 工具中心（包含 SRS 需求提取任务接口）。
|
||||
|
||||
## 开发与测试
|
||||
### 前端类型检查
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm exec tsc --noEmit
|
||||
```
|
||||
|
||||
### 后端测试
|
||||
```bash
|
||||
cd backend
|
||||
python -m pytest tests/test_testing_pipeline.py
|
||||
```
|
||||
|
||||
### 数据库迁移
|
||||
```bash
|
||||
cd backend
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
## 常见问题
|
||||
### 1) 提示某张表不存在（例如 tool_jobs）
|
||||
原因：数据库迁移未执行到最新版本。
|
||||
|
||||
处理：执行 alembic upgrade head，或重启后端让启动迁移自动执行。
|
||||
|
||||
### 2) 知识库文档处理失败并出现向量模型错误
|
||||
原因：模型配置与 provider 兼容性不匹配，或账号配额不足。
|
||||
|
||||
处理：
|
||||
- 检查 EMBEDDINGS_PROVIDER 与模型名是否匹配。
|
||||
- DashScope 兼容模式优先使用 text-embedding-v4。
|
||||
- 若报配额不足（AllocationQuota.FreeTierOnly），请切换可用 API Key 或开通付费资源。
|
||||
|
||||
### 3) 重复点击“开始处理”后状态异常
|
||||
后端已增加幂等处理与重试兜底，仍建议单次提交后等待任务轮询完成再重复操作。
|
||||
|
||||
## 许可证
|
||||
本项目基于 LICENSE 文件中定义的条款发布。
|
||||
55
rag-web-ui/.env.example
Normal file
55
rag-web-ui/.env.example
Normal file
@@ -0,0 +1,55 @@
|
||||
PROJECT_NAME=RAG Agent
|
||||
VERSION=0.1.0
|
||||
API_V1_STR=/api
|
||||
|
||||
MYSQL_SERVER=localhost
|
||||
MYSQL_PORT=3306
|
||||
MYSQL_USER=ragagent
|
||||
MYSQL_PASSWORD=ragagent
|
||||
MYSQL_DATABASE=ragagent
|
||||
|
||||
API_KEY= # API Key，服务商官网获取
|
||||
|
||||
OPENAI_API_KEY= # API Key，和上面一致
|
||||
OPENAI_API_BASE= # base-url，服务商官网获取
|
||||
OPENAI_MODEL= # 文本生成模型名称，例如 qwen3.5-plus
|
||||
OPENAI_EMBEDDINGS_MODEL= # 向量模型名称，例如 text-embedding-v4
|
||||
|
||||
SECRET_KEY=dev-secret-key-change-me
|
||||
ACCESS_TOKEN_EXPIRE_MINUTES=10080
|
||||
|
||||
CHAT_PROVIDER=dashscope
|
||||
EMBEDDINGS_PROVIDER=dashscope
|
||||
|
||||
DASH_SCOPE_API_KEY= # API Key，和上面一致
|
||||
DASH_SCOPE_API_BASE= # base-url，和上面一致
|
||||
DASH_SCOPE_CHAT_MODEL= # 文本生成模型名称
|
||||
DASH_SCOPE_EMBEDDINGS_MODEL= # 向量模型名称
|
||||
|
||||
VECTOR_STORE_TYPE=chroma
|
||||
CHROMA_DB_HOST=localhost
|
||||
CHROMA_DB_PORT=8001
|
||||
|
||||
MINIO_ENDPOINT=localhost:9000
|
||||
MINIO_ACCESS_KEY=minioadmin
|
||||
MINIO_SECRET_KEY=minioadmin
|
||||
MINIO_BUCKET_NAME=documents
|
||||
|
||||
RERANKER_API_URL= # reranker 模型的 base-url，服务商官网获取
|
||||
RERANKER_API_KEY= # API Key，和上面一致
|
||||
RERANKER_MODEL= # reranker 模型名称，例如 qwen3-vl-rerank
|
||||
RERANKER_TIMEOUT_SECONDS=10
|
||||
RERANKER_WEIGHT=0.75
|
||||
|
||||
GRAPHRAG_ENABLED=true
|
||||
GRAPHRAG_WORKING_DIR=./graphrag_cache
|
||||
GRAPHRAG_GRAPH_STORAGE=neo4j
|
||||
GRAPHRAG_QUERY_LEVEL=2
|
||||
GRAPHRAG_LOCAL_TOP_K=20
|
||||
GRAPHRAG_ENTITY_EXTRACT_MAX_GLEANING=1
|
||||
GRAPHRAG_EMBEDDING_DIM=1024
|
||||
GRAPHRAG_EMBEDDING_MAX_TOKEN_SIZE=8192
|
||||
|
||||
NEO4J_URL=bolt://localhost:7687
|
||||
NEO4J_USERNAME=neo4j
|
||||
NEO4J_PASSWORD=12345678
|
||||
27
rag-web-ui/.gitattributes
vendored
Normal file
27
rag-web-ui/.gitattributes
vendored
Normal file
@@ -0,0 +1,27 @@
|
||||
# Set default behavior to automatically normalize line endings
|
||||
* text=auto
|
||||
|
||||
# Unix/Linux/macOS style files (using LF)
|
||||
*.sh text eol=lf
|
||||
*.bash text eol=lf
|
||||
Dockerfile text eol=lf
|
||||
.dockerignore text eol=lf
|
||||
docker-compose*.yml text eol=lf
|
||||
*.py text eol=lf
|
||||
*.json text eol=lf
|
||||
*.yml text eol=lf
|
||||
*.yaml text eol=lf
|
||||
*.md text eol=lf
|
||||
|
||||
# Windows style files (using CRLF)
|
||||
*.{cmd,[cC][mM][dD]} text eol=crlf
|
||||
*.{bat,[bB][aA][tT]} text eol=crlf
|
||||
*.ps1 text eol=crlf
|
||||
|
||||
# Binary files (no conversion)
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.gif binary
|
||||
*.ico binary
|
||||
*.zip binary
|
||||
*.pdf binary
|
||||
60
rag-web-ui/.gitignore
vendored
Normal file
60
rag-web-ui/.gitignore
vendored
Normal file
@@ -0,0 +1,60 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
chroma_db/
|
||||
|
||||
# Node/Next.js
|
||||
node_modules/
|
||||
.next/
|
||||
out/
|
||||
.DS_Store
|
||||
*.pem
|
||||
.env.local
|
||||
.env.development.local
|
||||
.env.test.local
|
||||
.env.production.local
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
.vercel
|
||||
.turbo
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Project specific
|
||||
backend/static/
|
||||
backend/media/
|
||||
frontend/.env
|
||||
backend/.env
|
||||
201
rag-web-ui/LICENSE
Normal file
201
rag-web-ui/LICENSE
Normal file
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2024 Marcus Schiesser
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
35
rag-web-ui/backend/Dockerfile
Normal file
35
rag-web-ui/backend/Dockerfile
Normal file
@@ -0,0 +1,35 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
build-essential \
|
||||
default-libmysqlclient-dev \
|
||||
pkg-config \
|
||||
netcat-traditional \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy requirements file
|
||||
COPY requirements.txt .
|
||||
|
||||
# Install Python packages
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy entrypoint script first
|
||||
COPY entrypoint.sh .
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Copy application files
|
||||
COPY . .
|
||||
|
||||
# Create uploads directory
|
||||
RUN mkdir -p uploads
|
||||
|
||||
# Set Python path and environment
|
||||
ENV PYTHONPATH=/app
|
||||
ENV ENVIRONMENT=production
|
||||
|
||||
# Run the application
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
37
rag-web-ui/backend/Dockerfile.dev
Normal file
37
rag-web-ui/backend/Dockerfile.dev
Normal file
@@ -0,0 +1,37 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
build-essential \
|
||||
default-libmysqlclient-dev \
|
||||
pkg-config \
|
||||
netcat-traditional \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy requirements file
|
||||
COPY requirements.txt .
|
||||
|
||||
# Install Python packages with retry mechanism
|
||||
RUN pip install --no-cache-dir -r requirements.txt || \
|
||||
(echo "Retrying in 5 seconds..." && sleep 5 && pip install --no-cache-dir -r requirements.txt) || \
|
||||
(echo "Retrying in 10 seconds..." && sleep 10 && pip install --no-cache-dir -r requirements.txt)
|
||||
|
||||
# Copy entrypoint script first
|
||||
COPY entrypoint.sh .
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Copy the rest of the application
|
||||
COPY . .
|
||||
|
||||
# Create uploads directory
|
||||
RUN mkdir -p uploads
|
||||
|
||||
# Set Python path and environment
|
||||
ENV PYTHONPATH=/app
|
||||
ENV ENVIRONMENT=development
|
||||
|
||||
# Run the application
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
1
rag-web-ui/backend/__init__.py
Normal file
1
rag-web-ui/backend/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
|
||||
42
rag-web-ui/backend/alembic.ini
Normal file
42
rag-web-ui/backend/alembic.ini
Normal file
@@ -0,0 +1,42 @@
|
||||
[alembic]
|
||||
script_location = alembic
|
||||
sqlalchemy.url = mysql+mysqlconnector://ragagent:ragagent@db/ragagent
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic,uvicorn
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[logger_uvicorn]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = uvicorn
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
90
rag-web-ui/backend/alembic/env.py
Normal file
90
rag-web-ui/backend/alembic/env.py
Normal file
@@ -0,0 +1,90 @@
|
||||
import os
|
||||
import sys
|
||||
from logging.config import fileConfig
|
||||
from sqlalchemy import engine_from_config
|
||||
from sqlalchemy import pool
|
||||
from alembic import context
|
||||
|
||||
# Add the parent directory to Python path
|
||||
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from app.models.base import Base
|
||||
from app.models.user import User
|
||||
from app.models.knowledge import KnowledgeBase, Document
|
||||
from app.models.chat import Chat, Message
|
||||
from app.models.tooling import ToolJob, SRSExtraction, SRSRequirement
|
||||
from app.core.config import settings
|
||||
|
||||
# this is the Alembic Config object, which provides
|
||||
# access to the values within the .ini file in use.
|
||||
config = context.config
|
||||
|
||||
# Interpret the config file for Python logging.
|
||||
# This line sets up loggers basically.
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
# add your model's MetaData object here
|
||||
# for 'autogenerate' support
|
||||
target_metadata = Base.metadata
|
||||
|
||||
# other values from the config, defined by the needs of env.py,
|
||||
# can be acquired:
|
||||
# my_important_option = config.get_main_option("my_important_option")
|
||||
# ... etc.
|
||||
|
||||
def get_url():
|
||||
return settings.get_database_url
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode.
|
||||
|
||||
This configures the context with just a URL
|
||||
and not an Engine, though an Engine is acceptable
|
||||
here as well. By skipping the Engine creation
|
||||
we don't even need a DBAPI to be available.
|
||||
|
||||
Calls to context.execute() here emit the given string to the
|
||||
script output.
|
||||
|
||||
"""
|
||||
url = get_url()
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode.
|
||||
|
||||
In this scenario we need to create an Engine
|
||||
and associate a connection with the context.
|
||||
|
||||
"""
|
||||
configuration = config.get_section(config.config_ini_section)
|
||||
configuration["sqlalchemy.url"] = get_url()
|
||||
connectable = engine_from_config(
|
||||
configuration,
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection, target_metadata=target_metadata
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
26
rag-web-ui/backend/alembic/script.py.mako
Normal file
26
rag-web-ui/backend/alembic/script.py.mako
Normal file
@@ -0,0 +1,26 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, None] = ${repr(down_revision)}
|
||||
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,36 @@
|
||||
"""increase_api_key_length
|
||||
|
||||
Revision ID: 3580c0dcd005
|
||||
Revises: e214adf7fb66
|
||||
Create Date: 2024-01-20 14:25:00.000000
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '3580c0dcd005'
|
||||
down_revision: Union[str, None] = 'e214adf7fb66'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.alter_column('api_keys', 'key',
|
||||
existing_type=sa.String(length=64),
|
||||
type_=sa.String(length=128),
|
||||
existing_nullable=False)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.alter_column('api_keys', 'key',
|
||||
existing_type=sa.String(length=128),
|
||||
type_=sa.String(length=64),
|
||||
existing_nullable=False)
|
||||
# ### end Alembic commands ###
|
||||
@@ -0,0 +1,116 @@
|
||||
"""rename_metadata_to_chunk_metadata
|
||||
|
||||
Revision ID: 59cfa0f1361d
|
||||
Revises: initial_schema
|
||||
Create Date: 2025-01-13 23:26:38.232326
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import mysql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '59cfa0f1361d'
|
||||
down_revision: Union[str, None] = 'initial_schema'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_index(op.f('ix_chats_id'), 'chats', ['id'], unique=False)
|
||||
op.add_column('document_chunks', sa.Column('document_id', sa.Integer(), nullable=False))
|
||||
op.add_column('document_chunks', sa.Column('chunk_metadata', sa.JSON(), nullable=True))
|
||||
op.alter_column('document_chunks', 'created_at',
|
||||
existing_type=mysql.TIMESTAMP(),
|
||||
type_=sa.DateTime(),
|
||||
nullable=False,
|
||||
existing_server_default=sa.text('CURRENT_TIMESTAMP'))
|
||||
op.alter_column('document_chunks', 'updated_at',
|
||||
existing_type=mysql.TIMESTAMP(),
|
||||
type_=sa.DateTime(),
|
||||
nullable=False,
|
||||
existing_server_default=sa.text('CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP'))
|
||||
op.drop_index('idx_hash', table_name='document_chunks')
|
||||
op.create_index(op.f('ix_document_chunks_hash'), 'document_chunks', ['hash'], unique=False)
|
||||
op.create_foreign_key(None, 'document_chunks', 'knowledge_bases', ['kb_id'], ['id'])
|
||||
op.create_foreign_key(None, 'document_chunks', 'documents', ['document_id'], ['id'])
|
||||
op.drop_column('document_chunks', 'metadata')
|
||||
op.drop_index('idx_file_hash', table_name='documents')
|
||||
op.create_index(op.f('ix_documents_file_hash'), 'documents', ['file_hash'], unique=False)
|
||||
op.create_index(op.f('ix_documents_id'), 'documents', ['id'], unique=False)
|
||||
op.create_index(op.f('ix_knowledge_bases_id'), 'knowledge_bases', ['id'], unique=False)
|
||||
op.create_index(op.f('ix_messages_id'), 'messages', ['id'], unique=False)
|
||||
op.alter_column('processing_tasks', 'knowledge_base_id',
|
||||
existing_type=mysql.INTEGER(),
|
||||
nullable=True)
|
||||
op.alter_column('processing_tasks', 'document_id',
|
||||
existing_type=mysql.INTEGER(),
|
||||
nullable=True)
|
||||
op.alter_column('processing_tasks', 'status',
|
||||
existing_type=mysql.VARCHAR(length=50),
|
||||
nullable=True)
|
||||
op.alter_column('processing_tasks', 'created_at',
|
||||
existing_type=mysql.DATETIME(),
|
||||
nullable=True)
|
||||
op.alter_column('processing_tasks', 'updated_at',
|
||||
existing_type=mysql.DATETIME(),
|
||||
nullable=True)
|
||||
op.create_index(op.f('ix_processing_tasks_id'), 'processing_tasks', ['id'], unique=False)
|
||||
op.drop_index('email', table_name='users')
|
||||
op.drop_index('username', table_name='users')
|
||||
op.create_index(op.f('ix_users_email'), 'users', ['email'], unique=True)
|
||||
op.create_index(op.f('ix_users_id'), 'users', ['id'], unique=False)
|
||||
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_index(op.f('ix_users_username'), table_name='users')
|
||||
op.drop_index(op.f('ix_users_id'), table_name='users')
|
||||
op.drop_index(op.f('ix_users_email'), table_name='users')
|
||||
op.create_index('username', 'users', ['username'], unique=True)
|
||||
op.create_index('email', 'users', ['email'], unique=True)
|
||||
op.drop_index(op.f('ix_processing_tasks_id'), table_name='processing_tasks')
|
||||
op.alter_column('processing_tasks', 'updated_at',
|
||||
existing_type=mysql.DATETIME(),
|
||||
nullable=False)
|
||||
op.alter_column('processing_tasks', 'created_at',
|
||||
existing_type=mysql.DATETIME(),
|
||||
nullable=False)
|
||||
op.alter_column('processing_tasks', 'status',
|
||||
existing_type=mysql.VARCHAR(length=50),
|
||||
nullable=False)
|
||||
op.alter_column('processing_tasks', 'document_id',
|
||||
existing_type=mysql.INTEGER(),
|
||||
nullable=False)
|
||||
op.alter_column('processing_tasks', 'knowledge_base_id',
|
||||
existing_type=mysql.INTEGER(),
|
||||
nullable=False)
|
||||
op.drop_index(op.f('ix_messages_id'), table_name='messages')
|
||||
op.drop_index(op.f('ix_knowledge_bases_id'), table_name='knowledge_bases')
|
||||
op.drop_index(op.f('ix_documents_id'), table_name='documents')
|
||||
op.drop_index(op.f('ix_documents_file_hash'), table_name='documents')
|
||||
op.create_index('idx_file_hash', 'documents', ['file_hash'], unique=False)
|
||||
op.add_column('document_chunks', sa.Column('metadata', mysql.JSON(), nullable=True))
|
||||
op.drop_constraint(None, 'document_chunks', type_='foreignkey')
|
||||
op.drop_constraint(None, 'document_chunks', type_='foreignkey')
|
||||
op.drop_index(op.f('ix_document_chunks_hash'), table_name='document_chunks')
|
||||
op.create_index('idx_hash', 'document_chunks', ['hash'], unique=False)
|
||||
op.alter_column('document_chunks', 'updated_at',
|
||||
existing_type=sa.DateTime(),
|
||||
type_=mysql.TIMESTAMP(),
|
||||
nullable=True,
|
||||
existing_server_default=sa.text('CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP'))
|
||||
op.alter_column('document_chunks', 'created_at',
|
||||
existing_type=sa.DateTime(),
|
||||
type_=mysql.TIMESTAMP(),
|
||||
nullable=True,
|
||||
existing_server_default=sa.text('CURRENT_TIMESTAMP'))
|
||||
op.drop_column('document_chunks', 'chunk_metadata')
|
||||
op.drop_column('document_chunks', 'document_id')
|
||||
op.drop_index(op.f('ix_chats_id'), table_name='chats')
|
||||
# ### end Alembic commands ###
|
||||
@@ -0,0 +1,37 @@
|
||||
"""add_document_upload_id_to_processing_tasks
|
||||
|
||||
Revision ID: 5be054bd6587
|
||||
Revises: fd73eebc87c1
|
||||
Create Date: 2025-01-14 01:17:24.164593
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '5be054bd6587'
|
||||
down_revision: Union[str, None] = 'fd73eebc87c1'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# 1. 添加 document_upload_id 字段
|
||||
op.execute("""
|
||||
ALTER TABLE processing_tasks
|
||||
ADD COLUMN document_upload_id INT,
|
||||
ADD CONSTRAINT processing_tasks_document_upload_id_fkey
|
||||
FOREIGN KEY (document_upload_id) REFERENCES document_uploads(id)
|
||||
""")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# 1. 删除外键约束和字段
|
||||
op.execute("""
|
||||
ALTER TABLE processing_tasks
|
||||
DROP FOREIGN KEY processing_tasks_document_upload_id_fkey,
|
||||
DROP COLUMN document_upload_id
|
||||
""")
|
||||
@@ -0,0 +1,102 @@
|
||||
"""add tool jobs and srs tables
|
||||
|
||||
Revision ID: a4f9c89b7d11
|
||||
Revises: 3580c0dcd005
|
||||
Create Date: 2026-04-12 18:00:00.000000
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import mysql
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "a4f9c89b7d11"
|
||||
down_revision: Union[str, None] = "3580c0dcd005"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"tool_jobs",
|
||||
sa.Column("id", sa.Integer(), nullable=False),
|
||||
sa.Column("user_id", sa.Integer(), nullable=False),
|
||||
sa.Column("tool_name", sa.String(length=128), nullable=False),
|
||||
sa.Column("status", sa.String(length=32), nullable=False, server_default="pending"),
|
||||
sa.Column("input_file_name", sa.String(length=255), nullable=False),
|
||||
sa.Column("input_file_path", sa.String(length=512), nullable=False),
|
||||
sa.Column("error_message", sa.Text(), nullable=True),
|
||||
sa.Column("started_at", sa.DateTime(), nullable=True),
|
||||
sa.Column("completed_at", sa.DateTime(), nullable=True),
|
||||
sa.Column("output_summary", sa.JSON(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(["user_id"], ["users.id"]),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index(op.f("ix_tool_jobs_id"), "tool_jobs", ["id"], unique=False)
|
||||
op.create_index(op.f("ix_tool_jobs_tool_name"), "tool_jobs", ["tool_name"], unique=False)
|
||||
op.create_index(op.f("ix_tool_jobs_user_id"), "tool_jobs", ["user_id"], unique=False)
|
||||
|
||||
op.create_table(
|
||||
"srs_extractions",
|
||||
sa.Column("id", sa.Integer(), nullable=False),
|
||||
sa.Column("job_id", sa.Integer(), nullable=False),
|
||||
sa.Column("document_name", sa.String(length=255), nullable=False),
|
||||
sa.Column("document_title", sa.String(length=255), nullable=False),
|
||||
sa.Column("generated_at", sa.DateTime(), nullable=False),
|
||||
sa.Column("total_requirements", sa.Integer(), nullable=False),
|
||||
sa.Column("statistics", sa.JSON(), nullable=True),
|
||||
sa.Column("raw_output", sa.JSON(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(["job_id"], ["tool_jobs.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("job_id"),
|
||||
)
|
||||
op.create_index(op.f("ix_srs_extractions_id"), "srs_extractions", ["id"], unique=False)
|
||||
|
||||
op.create_table(
|
||||
"srs_requirements",
|
||||
sa.Column("id", sa.Integer(), nullable=False),
|
||||
sa.Column("extraction_id", sa.Integer(), nullable=False),
|
||||
sa.Column("requirement_uid", sa.String(length=64), nullable=False),
|
||||
sa.Column("title", sa.String(length=255), nullable=False),
|
||||
sa.Column("description", mysql.LONGTEXT(), nullable=False),
|
||||
sa.Column("priority", sa.String(length=16), nullable=False),
|
||||
sa.Column("acceptance_criteria", sa.JSON(), nullable=False),
|
||||
sa.Column("source_field", sa.String(length=255), nullable=False),
|
||||
sa.Column("section_number", sa.String(length=64), nullable=True),
|
||||
sa.Column("section_title", sa.String(length=255), nullable=True),
|
||||
sa.Column("requirement_type", sa.String(length=64), nullable=True),
|
||||
sa.Column("sort_order", sa.Integer(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(["extraction_id"], ["srs_extractions.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("extraction_id", "requirement_uid", name="uq_srs_extraction_requirement_uid"),
|
||||
)
|
||||
op.create_index(op.f("ix_srs_requirements_id"), "srs_requirements", ["id"], unique=False)
|
||||
op.create_index(
|
||||
"idx_srs_requirements_extraction_sort",
|
||||
"srs_requirements",
|
||||
["extraction_id", "sort_order"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_srs_requirements_extraction_sort", table_name="srs_requirements")
|
||||
op.drop_index(op.f("ix_srs_requirements_id"), table_name="srs_requirements")
|
||||
op.drop_table("srs_requirements")
|
||||
|
||||
op.drop_index(op.f("ix_srs_extractions_id"), table_name="srs_extractions")
|
||||
op.drop_table("srs_extractions")
|
||||
|
||||
op.drop_index(op.f("ix_tool_jobs_user_id"), table_name="tool_jobs")
|
||||
op.drop_index(op.f("ix_tool_jobs_tool_name"), table_name="tool_jobs")
|
||||
op.drop_index(op.f("ix_tool_jobs_id"), table_name="tool_jobs")
|
||||
op.drop_table("tool_jobs")
|
||||
@@ -0,0 +1,49 @@
|
||||
"""add_api_keys_table
|
||||
|
||||
Revision ID: e214adf7fb66
|
||||
Revises: 5be054bd6587
|
||||
Create Date: 2024-01-20 13:24:00.000000
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = 'e214adf7fb66'
|
||||
down_revision: Union[str, None] = '5be054bd6587'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table(
|
||||
'api_keys',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('key', sa.String(length=64), nullable=False),
|
||||
sa.Column('name', sa.String(length=255), nullable=False),
|
||||
sa.Column('user_id', sa.Integer(), nullable=False),
|
||||
sa.Column('is_active', sa.Boolean(), nullable=False, default=True),
|
||||
sa.Column('last_used_at', sa.DateTime(), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False, server_default=sa.text('CURRENT_TIMESTAMP')),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False, server_default=sa.text('CURRENT_TIMESTAMP'), onupdate=sa.text('CURRENT_TIMESTAMP')),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('key')
|
||||
)
|
||||
op.create_index(op.f('ix_api_keys_id'), 'api_keys', ['id'], unique=False)
|
||||
op.create_index(op.f('ix_api_keys_key'), 'api_keys', ['key'], unique=True)
|
||||
op.create_index(op.f('ix_api_keys_name'), 'api_keys', ['name'], unique=False)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_index(op.f('ix_api_keys_name'), table_name='api_keys')
|
||||
op.drop_index(op.f('ix_api_keys_key'), table_name='api_keys')
|
||||
op.drop_index(op.f('ix_api_keys_id'), table_name='api_keys')
|
||||
op.drop_table('api_keys')
|
||||
# ### end Alembic commands ###
|
||||
@@ -0,0 +1,44 @@
|
||||
"""add document uploads table
|
||||
|
||||
Revision ID: fd73eebc87c1
|
||||
Revises: 59cfa0f1361d
|
||||
Create Date: 2024-01-13 16:24:07.182834
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'fd73eebc87c1'
|
||||
down_revision = '59cfa0f1361d'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
'document_uploads',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('knowledge_base_id', sa.Integer(), nullable=False),
|
||||
sa.Column('file_name', sa.String(255), nullable=False),
|
||||
sa.Column('file_hash', sa.String(64), nullable=False),
|
||||
sa.Column('file_size', sa.BigInteger(), nullable=False),
|
||||
sa.Column('content_type', sa.String(100), nullable=False),
|
||||
sa.Column('temp_path', sa.String(255), nullable=False),
|
||||
sa.Column('created_at', sa.TIMESTAMP(), server_default=sa.text('now()'), nullable=False),
|
||||
sa.Column('status', sa.String(50), nullable=False, server_default='pending'),
|
||||
sa.Column('error_message', sa.Text(), nullable=True),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.ForeignKeyConstraint(['knowledge_base_id'], ['knowledge_bases.id'], ondelete='CASCADE')
|
||||
)
|
||||
|
||||
# 添加索引以加速查询
|
||||
op.create_index('ix_document_uploads_created_at', 'document_uploads', ['created_at'])
|
||||
op.create_index('ix_document_uploads_status', 'document_uploads', ['status'])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index('ix_document_uploads_status')
|
||||
op.drop_index('ix_document_uploads_created_at')
|
||||
op.drop_table('document_uploads')
|
||||
148
rag-web-ui/backend/alembic/versions/initial_schema.py
Normal file
148
rag-web-ui/backend/alembic/versions/initial_schema.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""initial schema
|
||||
|
||||
Revision ID: initial_schema
|
||||
Revises:
|
||||
Create Date: 2024-01-13 15:00:00.000000
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import mysql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = 'initial_schema'
|
||||
down_revision: Union[str, None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
def upgrade() -> None:
|
||||
# Create users table
|
||||
op.create_table(
|
||||
'users',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('email', sa.String(255), nullable=False),
|
||||
sa.Column('username', sa.String(255), nullable=False),
|
||||
sa.Column('hashed_password', sa.String(255), nullable=False),
|
||||
sa.Column('is_active', sa.Boolean(), nullable=True, default=True),
|
||||
sa.Column('is_superuser', sa.Boolean(), nullable=True, default=False),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('email'),
|
||||
sa.UniqueConstraint('username')
|
||||
)
|
||||
|
||||
# Create knowledge_bases table
|
||||
op.create_table(
|
||||
'knowledge_bases',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('name', sa.String(255), nullable=False),
|
||||
sa.Column('description', mysql.LONGTEXT(), nullable=True),
|
||||
sa.Column('user_id', sa.Integer(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
|
||||
# Create documents table
|
||||
op.create_table(
|
||||
'documents',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('file_path', sa.String(255), nullable=False),
|
||||
sa.Column('file_name', sa.String(255), nullable=False),
|
||||
sa.Column('file_size', sa.Integer(), nullable=True),
|
||||
sa.Column('content_type', sa.String(100), nullable=True),
|
||||
sa.Column('file_hash', sa.String(64), nullable=True),
|
||||
sa.Column('knowledge_base_id', sa.Integer(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['knowledge_base_id'], ['knowledge_bases.id'], ),
|
||||
sa.PrimaryKeyConstraint('id'),
|
||||
sa.UniqueConstraint('knowledge_base_id', 'file_name', name='uq_kb_file_name')
|
||||
)
|
||||
|
||||
# Create document_chunks table
|
||||
op.create_table(
|
||||
'document_chunks',
|
||||
sa.Column('id', sa.String(64), nullable=False),
|
||||
sa.Column('kb_id', sa.Integer(), nullable=False),
|
||||
sa.Column('file_name', sa.String(255), nullable=False),
|
||||
sa.Column('metadata', sa.JSON(), nullable=True),
|
||||
sa.Column('hash', sa.String(64), nullable=False),
|
||||
sa.Column('created_at', sa.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP')),
|
||||
sa.Column('updated_at', sa.TIMESTAMP(), server_default=sa.text('CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP')),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
|
||||
# Create chats table
|
||||
op.create_table(
|
||||
'chats',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('title', sa.String(255), nullable=False),
|
||||
sa.Column('user_id', sa.Integer(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
|
||||
# Create chat_knowledge_bases table (association table)
|
||||
op.create_table(
|
||||
'chat_knowledge_bases',
|
||||
sa.Column('chat_id', sa.Integer(), nullable=False),
|
||||
sa.Column('knowledge_base_id', sa.Integer(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ),
|
||||
sa.ForeignKeyConstraint(['knowledge_base_id'], ['knowledge_bases.id'], ),
|
||||
sa.PrimaryKeyConstraint('chat_id', 'knowledge_base_id')
|
||||
)
|
||||
|
||||
# Create messages table
|
||||
op.create_table(
|
||||
'messages',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('chat_id', sa.Integer(), nullable=False),
|
||||
sa.Column('role', sa.String(50), nullable=False),
|
||||
sa.Column('content', mysql.LONGTEXT(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
|
||||
# Create processing_tasks table
|
||||
op.create_table(
|
||||
'processing_tasks',
|
||||
sa.Column('id', sa.Integer(), nullable=False),
|
||||
sa.Column('knowledge_base_id', sa.Integer(), nullable=False),
|
||||
sa.Column('document_id', sa.Integer(), nullable=False),
|
||||
sa.Column('status', sa.String(50), nullable=False, default='pending'),
|
||||
sa.Column('error_message', sa.Text(), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(), nullable=False),
|
||||
sa.Column('updated_at', sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['document_id'], ['documents.id'], ),
|
||||
sa.ForeignKeyConstraint(['knowledge_base_id'], ['knowledge_bases.id'], ),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
|
||||
# Create indexes
|
||||
op.create_index('idx_kb_file_name', 'document_chunks', ['kb_id', 'file_name'])
|
||||
op.create_index('idx_hash', 'document_chunks', ['hash'])
|
||||
op.create_index('idx_file_hash', 'documents', ['file_hash'])
|
||||
|
||||
def downgrade() -> None:
|
||||
# Drop indexes
|
||||
op.drop_index('idx_hash', table_name='document_chunks')
|
||||
op.drop_index('idx_kb_file_name', table_name='document_chunks')
|
||||
op.drop_index('idx_file_hash', table_name='documents')
|
||||
|
||||
# Drop tables in reverse order
|
||||
op.drop_table('processing_tasks')
|
||||
op.drop_table('messages')
|
||||
op.drop_table('chat_knowledge_bases')
|
||||
op.drop_table('chats')
|
||||
op.drop_table('document_chunks')
|
||||
op.drop_table('documents')
|
||||
op.drop_table('knowledge_bases')
|
||||
op.drop_table('users')
|
||||
0
rag-web-ui/backend/app/__init__.py
Normal file
0
rag-web-ui/backend/app/__init__.py
Normal file
0
rag-web-ui/backend/app/api/__init__.py
Normal file
0
rag-web-ui/backend/app/api/__init__.py
Normal file
0
rag-web-ui/backend/app/api/api_v1/__init__.py
Normal file
0
rag-web-ui/backend/app/api/api_v1/__init__.py
Normal file
11
rag-web-ui/backend/app/api/api_v1/api.py
Normal file
11
rag-web-ui/backend/app/api/api_v1/api.py
Normal file
@@ -0,0 +1,11 @@
|
||||
from fastapi import APIRouter
|
||||
from app.api.api_v1 import api_keys, auth, chat, knowledge_base, testing, tools
|
||||
|
||||
api_router = APIRouter()
|
||||
|
||||
api_router.include_router(auth.router, prefix="/auth", tags=["auth"])
|
||||
api_router.include_router(knowledge_base.router, prefix="/knowledge-base", tags=["knowledge-base"])
|
||||
api_router.include_router(chat.router, prefix="/chat", tags=["chat"])
|
||||
api_router.include_router(api_keys.router, prefix="/api-keys", tags=["api-keys"])
|
||||
api_router.include_router(testing.router, prefix="/testing", tags=["testing"])
|
||||
api_router.include_router(tools.router, prefix="/tools", tags=["tools"])
|
||||
84
rag-web-ui/backend/app/api/api_v1/api_keys.py
Normal file
84
rag-web-ui/backend/app/api/api_v1/api_keys.py
Normal file
@@ -0,0 +1,84 @@
|
||||
from typing import Any, List
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
import logging
|
||||
|
||||
from app import models, schemas
|
||||
from app.db.session import get_db
|
||||
from app.services.api_key import APIKeyService
|
||||
from app.core.security import get_current_user
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@router.get("/", response_model=List[schemas.APIKey])
|
||||
def read_api_keys(
|
||||
db: Session = Depends(get_db),
|
||||
skip: int = 0,
|
||||
limit: int = 100,
|
||||
current_user: models.User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
"""
|
||||
Retrieve API keys.
|
||||
"""
|
||||
api_keys = APIKeyService.get_api_keys(
|
||||
db=db, user_id=current_user.id, skip=skip, limit=limit
|
||||
)
|
||||
return api_keys
|
||||
|
||||
@router.post("/", response_model=schemas.APIKey)
|
||||
def create_api_key(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
api_key_in: schemas.APIKeyCreate,
|
||||
current_user: models.User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
"""
|
||||
Create new API key.
|
||||
"""
|
||||
api_key = APIKeyService.create_api_key(
|
||||
db=db, user_id=current_user.id, name=api_key_in.name
|
||||
)
|
||||
logger.info(f"API key created: {api_key.key} for user {current_user.id}")
|
||||
return api_key
|
||||
|
||||
@router.put("/{id}", response_model=schemas.APIKey)
|
||||
def update_api_key(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
id: int,
|
||||
api_key_in: schemas.APIKeyUpdate,
|
||||
current_user: models.User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
"""
|
||||
Update API key.
|
||||
"""
|
||||
api_key = APIKeyService.get_api_key(db=db, api_key_id=id)
|
||||
if not api_key:
|
||||
raise HTTPException(status_code=404, detail="API key not found")
|
||||
if api_key.user_id != current_user.id:
|
||||
raise HTTPException(status_code=403, detail="Not enough permissions")
|
||||
|
||||
api_key = APIKeyService.update_api_key(db=db, api_key=api_key, update_data=api_key_in)
|
||||
logger.info(f"API key updated: {api_key.key} for user {current_user.id}")
|
||||
return api_key
|
||||
|
||||
@router.delete("/{id}", response_model=schemas.APIKey)
|
||||
def delete_api_key(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
id: int,
|
||||
current_user: models.User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
"""
|
||||
Delete API key.
|
||||
"""
|
||||
api_key = APIKeyService.get_api_key(db=db, api_key_id=id)
|
||||
if not api_key:
|
||||
raise HTTPException(status_code=404, detail="API key not found")
|
||||
if api_key.user_id != current_user.id:
|
||||
raise HTTPException(status_code=403, detail="Not enough permissions")
|
||||
|
||||
APIKeyService.delete_api_key(db=db, api_key=api_key)
|
||||
logger.info(f"API key deleted: {api_key.key} for user {current_user.id}")
|
||||
return api_key
|
||||
88
rag-web-ui/backend/app/api/api_v1/auth.py
Normal file
88
rag-web-ui/backend/app/api/api_v1/auth.py
Normal file
@@ -0,0 +1,88 @@
|
||||
from datetime import timedelta
|
||||
from typing import Any
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from fastapi.security import OAuth2PasswordRequestForm
|
||||
from sqlalchemy.orm import Session
|
||||
from requests.exceptions import RequestException
|
||||
|
||||
from app.core import security
|
||||
from app.core.security import get_current_user
|
||||
from app.core.config import settings
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.token import Token
|
||||
from app.schemas.user import UserCreate, UserResponse
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.post("/register", response_model=UserResponse)
|
||||
def register(*, db: Session = Depends(get_db), user_in: UserCreate) -> Any:
|
||||
"""
|
||||
Register a new user.
|
||||
"""
|
||||
try:
|
||||
# Check if user with this email exists
|
||||
user = db.query(User).filter(User.email == user_in.email).first()
|
||||
if user:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="A user with this email already exists.",
|
||||
)
|
||||
|
||||
# Check if user with this username exists
|
||||
user = db.query(User).filter(User.username == user_in.username).first()
|
||||
if user:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="A user with this username already exists.",
|
||||
)
|
||||
|
||||
# Create new user
|
||||
user = User(
|
||||
email=user_in.email,
|
||||
username=user_in.username,
|
||||
hashed_password=security.get_password_hash(user_in.password),
|
||||
)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
db.refresh(user)
|
||||
return user
|
||||
except RequestException as e:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Network error or server is unreachable. Please try again later.",
|
||||
) from e
|
||||
|
||||
@router.post("/token", response_model=Token)
|
||||
def login_access_token(
|
||||
db: Session = Depends(get_db), form_data: OAuth2PasswordRequestForm = Depends()
|
||||
) -> Any:
|
||||
"""
|
||||
OAuth2 compatible token login, get an access token for future requests.
|
||||
"""
|
||||
user = db.query(User).filter(User.username == form_data.username).first()
|
||||
if not user or not security.verify_password(form_data.password, user.hashed_password):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Incorrect username or password",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
elif not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Inactive user",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
access_token_expires = timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||
access_token = security.create_access_token(
|
||||
data={"sub": user.username}, expires_delta=access_token_expires
|
||||
)
|
||||
return {"access_token": access_token, "token_type": "bearer"}
|
||||
|
||||
@router.post("/test-token", response_model=UserResponse)
|
||||
def test_token(current_user: User = Depends(get_current_user)) -> Any:
|
||||
"""
|
||||
Test access token by getting current user.
|
||||
"""
|
||||
return current_user
|
||||
155
rag-web-ui/backend/app/api/api_v1/chat.py
Normal file
155
rag-web-ui/backend/app/api/api_v1/chat.py
Normal file
@@ -0,0 +1,155 @@
|
||||
from typing import List, Any
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from sqlalchemy.orm import Session, joinedload
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.models.chat import Chat, Message
|
||||
from app.models.knowledge import KnowledgeBase
|
||||
from app.schemas.chat import (
|
||||
ChatCreate,
|
||||
ChatResponse,
|
||||
ChatUpdate,
|
||||
MessageCreate,
|
||||
MessageResponse
|
||||
)
|
||||
from app.core.security import get_current_user
|
||||
from app.services.chat_service import generate_response
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.post("/", response_model=ChatResponse)
|
||||
def create_chat(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
chat_in: ChatCreate,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
# Verify knowledge bases exist and belong to user
|
||||
knowledge_bases = (
|
||||
db.query(KnowledgeBase)
|
||||
.filter(
|
||||
KnowledgeBase.id.in_(chat_in.knowledge_base_ids),
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
)
|
||||
.all()
|
||||
)
|
||||
if len(knowledge_bases) != len(chat_in.knowledge_base_ids):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="One or more knowledge bases not found"
|
||||
)
|
||||
|
||||
chat = Chat(
|
||||
title=chat_in.title,
|
||||
user_id=current_user.id,
|
||||
)
|
||||
chat.knowledge_bases = knowledge_bases
|
||||
|
||||
db.add(chat)
|
||||
db.commit()
|
||||
db.refresh(chat)
|
||||
return chat
|
||||
|
||||
@router.get("/", response_model=List[ChatResponse])
|
||||
def get_chats(
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
skip: int = 0,
|
||||
limit: int = 100
|
||||
) -> Any:
|
||||
chats = (
|
||||
db.query(Chat)
|
||||
.filter(Chat.user_id == current_user.id)
|
||||
.offset(skip)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return chats
|
||||
|
||||
@router.get("/{chat_id}", response_model=ChatResponse)
|
||||
def get_chat(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
chat_id: int,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
chat = (
|
||||
db.query(Chat)
|
||||
.filter(
|
||||
Chat.id == chat_id,
|
||||
Chat.user_id == current_user.id
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if not chat:
|
||||
raise HTTPException(status_code=404, detail="Chat not found")
|
||||
return chat
|
||||
|
||||
@router.post("/{chat_id}/messages")
|
||||
async def create_message(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
chat_id: int,
|
||||
messages: dict,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> StreamingResponse:
|
||||
chat = (
|
||||
db.query(Chat)
|
||||
.options(joinedload(Chat.knowledge_bases))
|
||||
.filter(
|
||||
Chat.id == chat_id,
|
||||
Chat.user_id == current_user.id
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if not chat:
|
||||
raise HTTPException(status_code=404, detail="Chat not found")
|
||||
|
||||
# Get the last user message
|
||||
last_message = messages["messages"][-1]
|
||||
if last_message["role"] != "user":
|
||||
raise HTTPException(status_code=400, detail="Last message must be from user")
|
||||
|
||||
# Get knowledge base IDs
|
||||
knowledge_base_ids = [kb.id for kb in chat.knowledge_bases]
|
||||
|
||||
async def response_stream():
|
||||
async for chunk in generate_response(
|
||||
query=last_message["content"],
|
||||
messages=messages,
|
||||
knowledge_base_ids=knowledge_base_ids,
|
||||
chat_id=chat_id,
|
||||
db=db
|
||||
):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(
|
||||
response_stream(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"x-vercel-ai-data-stream": "v1"
|
||||
}
|
||||
)
|
||||
|
||||
@router.delete("/{chat_id}")
|
||||
def delete_chat(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
chat_id: int,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
chat = (
|
||||
db.query(Chat)
|
||||
.filter(
|
||||
Chat.id == chat_id,
|
||||
Chat.user_id == current_user.id
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if not chat:
|
||||
raise HTTPException(status_code=404, detail="Chat not found")
|
||||
|
||||
db.delete(chat)
|
||||
db.commit()
|
||||
return {"status": "success"}
|
||||
575
rag-web-ui/backend/app/api/api_v1/knowledge_base.py
Normal file
575
rag-web-ui/backend/app/api/api_v1/knowledge_base.py
Normal file
@@ -0,0 +1,575 @@
|
||||
import hashlib
|
||||
from typing import List, Any, Dict
|
||||
from fastapi import APIRouter, Depends, HTTPException, UploadFile, File, BackgroundTasks, Query
|
||||
from sqlalchemy.orm import Session
|
||||
from langchain_chroma import Chroma
|
||||
from sqlalchemy import text
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm import selectinload
|
||||
import time
|
||||
import asyncio
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.core.security import get_current_user
|
||||
from app.models.knowledge import KnowledgeBase, Document, ProcessingTask, DocumentChunk, DocumentUpload
|
||||
from app.schemas.knowledge import (
|
||||
KnowledgeBaseCreate,
|
||||
KnowledgeBaseResponse,
|
||||
KnowledgeBaseUpdate,
|
||||
DocumentResponse,
|
||||
PreviewRequest
|
||||
)
|
||||
from app.services.document_processor import process_document_background, upload_document, preview_document, PreviewResult
|
||||
from app.core.config import settings
|
||||
from app.core.minio import get_minio_client
|
||||
from minio.error import MinioException
|
||||
from app.services.vector_store import VectorStoreFactory
|
||||
from app.services.embedding.embedding_factory import EmbeddingsFactory
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class TestRetrievalRequest(BaseModel):
|
||||
query: str
|
||||
kb_id: int
|
||||
top_k: int
|
||||
|
||||
@router.post("", response_model=KnowledgeBaseResponse)
|
||||
def create_knowledge_base(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
kb_in: KnowledgeBaseCreate,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
"""
|
||||
Create new knowledge base.
|
||||
"""
|
||||
kb = KnowledgeBase(
|
||||
name=kb_in.name,
|
||||
description=kb_in.description,
|
||||
user_id=current_user.id
|
||||
)
|
||||
db.add(kb)
|
||||
db.commit()
|
||||
db.refresh(kb)
|
||||
logger.info(f"Knowledge base created: {kb.name} for user {current_user.id}")
|
||||
return kb
|
||||
|
||||
@router.get("", response_model=List[KnowledgeBaseResponse])
|
||||
def get_knowledge_bases(
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
skip: int = 0,
|
||||
limit: int = 100
|
||||
) -> Any:
|
||||
"""
|
||||
Retrieve knowledge bases.
|
||||
"""
|
||||
knowledge_bases = (
|
||||
db.query(KnowledgeBase)
|
||||
.filter(KnowledgeBase.user_id == current_user.id)
|
||||
.offset(skip)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return knowledge_bases
|
||||
|
||||
@router.get("/{kb_id}", response_model=KnowledgeBaseResponse)
|
||||
def get_knowledge_base(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
kb_id: int,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
"""
|
||||
Get knowledge base by ID.
|
||||
"""
|
||||
from sqlalchemy.orm import joinedload
|
||||
|
||||
kb = (
|
||||
db.query(KnowledgeBase)
|
||||
.options(
|
||||
joinedload(KnowledgeBase.documents)
|
||||
.joinedload(Document.processing_tasks)
|
||||
)
|
||||
.filter(
|
||||
KnowledgeBase.id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
)
|
||||
.first()
|
||||
)
|
||||
|
||||
if not kb:
|
||||
raise HTTPException(status_code=404, detail="Knowledge base not found")
|
||||
|
||||
return kb
|
||||
|
||||
@router.put("/{kb_id}", response_model=KnowledgeBaseResponse)
|
||||
def update_knowledge_base(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
kb_id: int,
|
||||
kb_in: KnowledgeBaseUpdate,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
"""
|
||||
Update knowledge base.
|
||||
"""
|
||||
kb = db.query(KnowledgeBase).filter(
|
||||
KnowledgeBase.id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if not kb:
|
||||
raise HTTPException(status_code=404, detail="Knowledge base not found")
|
||||
|
||||
for field, value in kb_in.dict(exclude_unset=True).items():
|
||||
setattr(kb, field, value)
|
||||
|
||||
db.add(kb)
|
||||
db.commit()
|
||||
db.refresh(kb)
|
||||
logger.info(f"Knowledge base updated: {kb.name} for user {current_user.id}")
|
||||
return kb
|
||||
|
||||
@router.delete("/{kb_id}")
|
||||
async def delete_knowledge_base(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
kb_id: int,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
"""
|
||||
Delete knowledge base and all associated resources.
|
||||
"""
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
kb = (
|
||||
db.query(KnowledgeBase)
|
||||
.filter(
|
||||
KnowledgeBase.id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if not kb:
|
||||
raise HTTPException(status_code=404, detail="Knowledge base not found")
|
||||
|
||||
try:
|
||||
# Get all document file paths before deletion
|
||||
document_paths = [doc.file_path for doc in kb.documents]
|
||||
|
||||
# Initialize services
|
||||
minio_client = get_minio_client()
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
|
||||
vector_store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{kb_id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
|
||||
# Clean up external resources first
|
||||
cleanup_errors = []
|
||||
|
||||
# 1. Clean up MinIO files
|
||||
try:
|
||||
# Delete all objects with prefix kb_{kb_id}/
|
||||
objects = minio_client.list_objects(settings.MINIO_BUCKET_NAME, prefix=f"kb_{kb_id}/")
|
||||
for obj in objects:
|
||||
minio_client.remove_object(settings.MINIO_BUCKET_NAME, obj.object_name)
|
||||
logger.info(f"Cleaned up MinIO files for knowledge base {kb_id}")
|
||||
except MinioException as e:
|
||||
cleanup_errors.append(f"Failed to clean up MinIO files: {str(e)}")
|
||||
logger.error(f"MinIO cleanup error for kb {kb_id}: {str(e)}")
|
||||
|
||||
# 2. Clean up vector store
|
||||
try:
|
||||
vector_store._store.delete_collection(f"kb_{kb_id}")
|
||||
logger.info(f"Cleaned up vector store for knowledge base {kb_id}")
|
||||
except Exception as e:
|
||||
cleanup_errors.append(f"Failed to clean up vector store: {str(e)}")
|
||||
logger.error(f"Vector store cleanup error for kb {kb_id}: {str(e)}")
|
||||
|
||||
# Finally, delete database records in a single transaction
|
||||
db.delete(kb)
|
||||
db.commit()
|
||||
|
||||
# Report any cleanup errors in the response
|
||||
if cleanup_errors:
|
||||
return {
|
||||
"message": "Knowledge base deleted with cleanup warnings",
|
||||
"warnings": cleanup_errors
|
||||
}
|
||||
|
||||
return {"message": "Knowledge base and all associated resources deleted successfully"}
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
logger.error(f"Failed to delete knowledge base {kb_id}: {str(e)}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to delete knowledge base: {str(e)}")
|
||||
|
||||
# Batch upload documents
|
||||
@router.post("/{kb_id}/documents/upload")
|
||||
async def upload_kb_documents(
|
||||
kb_id: int,
|
||||
files: List[UploadFile],
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""
|
||||
Upload multiple documents to MinIO.
|
||||
"""
|
||||
kb = db.query(KnowledgeBase).filter(
|
||||
KnowledgeBase.id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
if not kb:
|
||||
raise HTTPException(status_code=404, detail="Knowledge base not found")
|
||||
|
||||
results = []
|
||||
for file in files:
|
||||
# 1. 计算文件 hash
|
||||
file_content = await file.read()
|
||||
file_hash = hashlib.sha256(file_content).hexdigest()
|
||||
|
||||
# 2. 检查是否存在完全相同的文件（名称和hash都相同）
|
||||
existing_document = db.query(Document).filter(
|
||||
Document.file_name == file.filename,
|
||||
Document.file_hash == file_hash,
|
||||
Document.knowledge_base_id == kb_id
|
||||
).first()
|
||||
|
||||
if existing_document:
|
||||
# 完全相同的文件，直接返回
|
||||
results.append({
|
||||
"document_id": existing_document.id,
|
||||
"file_name": existing_document.file_name,
|
||||
"status": "exists",
|
||||
"message": "文件已存在且已处理完成",
|
||||
"skip_processing": True
|
||||
})
|
||||
continue
|
||||
|
||||
# 3. 上传到临时目录
|
||||
temp_path = f"kb_{kb_id}/temp/{file.filename}"
|
||||
await file.seek(0)
|
||||
try:
|
||||
minio_client = get_minio_client()
|
||||
file_size = len(file_content) # 使用之前读取的文件内容长度
|
||||
minio_client.put_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=temp_path,
|
||||
data=file.file,
|
||||
length=file_size, # 指定文件大小
|
||||
content_type=file.content_type
|
||||
)
|
||||
except MinioException as e:
|
||||
logger.error(f"Failed to upload file to MinIO: {str(e)}")
|
||||
raise HTTPException(status_code=500, detail="Failed to upload file")
|
||||
|
||||
# 4. 创建上传记录
|
||||
upload = DocumentUpload(
|
||||
knowledge_base_id=kb_id,
|
||||
file_name=file.filename,
|
||||
file_hash=file_hash,
|
||||
file_size=len(file_content),
|
||||
content_type=file.content_type,
|
||||
temp_path=temp_path
|
||||
)
|
||||
db.add(upload)
|
||||
db.commit()
|
||||
db.refresh(upload)
|
||||
|
||||
results.append({
|
||||
"upload_id": upload.id,
|
||||
"file_name": file.filename,
|
||||
"temp_path": temp_path,
|
||||
"status": "pending",
|
||||
"skip_processing": False
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
@router.post("/{kb_id}/documents/preview")
|
||||
async def preview_kb_documents(
|
||||
kb_id: int,
|
||||
preview_request: PreviewRequest,
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Dict[int, PreviewResult]:
|
||||
"""
|
||||
Preview multiple documents' chunks.
|
||||
"""
|
||||
results = {}
|
||||
for doc_id in preview_request.document_ids:
|
||||
document = db.query(Document).join(KnowledgeBase).filter(
|
||||
Document.id == doc_id,
|
||||
Document.knowledge_base_id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if document:
|
||||
file_path = document.file_path
|
||||
else:
|
||||
upload = db.query(DocumentUpload).join(KnowledgeBase).filter(
|
||||
DocumentUpload.id == doc_id,
|
||||
DocumentUpload.knowledge_base_id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if not upload:
|
||||
raise HTTPException(status_code=404, detail=f"Document {doc_id} not found")
|
||||
|
||||
file_path = upload.temp_path
|
||||
|
||||
preview = await preview_document(
|
||||
file_path,
|
||||
chunk_size=preview_request.chunk_size,
|
||||
chunk_overlap=preview_request.chunk_overlap
|
||||
)
|
||||
results[doc_id] = preview
|
||||
|
||||
return results
|
||||
|
||||
@router.post("/{kb_id}/documents/process")
|
||||
async def process_kb_documents(
|
||||
kb_id: int,
|
||||
upload_results: List[dict],
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""
|
||||
Process multiple documents asynchronously.
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
kb = db.query(KnowledgeBase).filter(
|
||||
KnowledgeBase.id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if not kb:
|
||||
raise HTTPException(status_code=404, detail="Knowledge base not found")
|
||||
|
||||
task_info = []
|
||||
upload_ids = []
|
||||
|
||||
for result in upload_results:
|
||||
if result.get("skip_processing"):
|
||||
continue
|
||||
upload_ids.append(result["upload_id"])
|
||||
|
||||
if not upload_ids:
|
||||
return {"tasks": []}
|
||||
|
||||
uploads = db.query(DocumentUpload).filter(DocumentUpload.id.in_(upload_ids)).all()
|
||||
uploads_dict = {upload.id: upload for upload in uploads}
|
||||
|
||||
all_tasks = []
|
||||
for upload_id in upload_ids:
|
||||
upload = uploads_dict.get(upload_id)
|
||||
if not upload:
|
||||
continue
|
||||
|
||||
task = ProcessingTask(
|
||||
document_upload_id=upload_id,
|
||||
knowledge_base_id=kb_id,
|
||||
status="pending"
|
||||
)
|
||||
all_tasks.append(task)
|
||||
|
||||
db.add_all(all_tasks)
|
||||
db.commit()
|
||||
|
||||
for task in all_tasks:
|
||||
db.refresh(task)
|
||||
|
||||
task_data = []
|
||||
for i, upload_id in enumerate(upload_ids):
|
||||
if i < len(all_tasks):
|
||||
task = all_tasks[i]
|
||||
upload = uploads_dict.get(upload_id)
|
||||
|
||||
task_info.append({
|
||||
"upload_id": upload_id,
|
||||
"task_id": task.id
|
||||
})
|
||||
|
||||
if upload:
|
||||
task_data.append({
|
||||
"task_id": task.id,
|
||||
"upload_id": upload_id,
|
||||
"temp_path": upload.temp_path,
|
||||
"file_name": upload.file_name
|
||||
})
|
||||
|
||||
background_tasks.add_task(
|
||||
add_processing_tasks_to_queue,
|
||||
task_data,
|
||||
kb_id
|
||||
)
|
||||
|
||||
return {"tasks": task_info}
|
||||
|
||||
async def add_processing_tasks_to_queue(task_data, kb_id):
|
||||
"""Helper function to add document processing tasks to the queue without blocking the main response."""
|
||||
for data in task_data:
|
||||
asyncio.create_task(
|
||||
process_document_background(
|
||||
data["temp_path"],
|
||||
data["file_name"],
|
||||
kb_id,
|
||||
data["task_id"],
|
||||
None
|
||||
)
|
||||
)
|
||||
logger.info(f"Added {len(task_data)} document processing tasks to queue")
|
||||
|
||||
@router.post("/cleanup")
|
||||
async def cleanup_temp_files(
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""
|
||||
Clean up expired temporary files.
|
||||
"""
|
||||
expired_time = datetime.utcnow() - timedelta(hours=24)
|
||||
expired_uploads = db.query(DocumentUpload).filter(
|
||||
DocumentUpload.created_at < expired_time
|
||||
).all()
|
||||
|
||||
minio_client = get_minio_client()
|
||||
for upload in expired_uploads:
|
||||
try:
|
||||
minio_client.remove_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=upload.temp_path
|
||||
)
|
||||
except MinioException as e:
|
||||
logger.error(f"Failed to delete temp file {upload.temp_path}: {str(e)}")
|
||||
|
||||
db.delete(upload)
|
||||
|
||||
db.commit()
|
||||
|
||||
return {"message": f"Cleaned up {len(expired_uploads)} expired uploads"}
|
||||
|
||||
@router.get("/{kb_id}/documents/tasks")
|
||||
async def get_processing_tasks(
|
||||
kb_id: int,
|
||||
task_ids: str = Query(..., description="Comma-separated list of task IDs to check status for"),
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
):
|
||||
"""
|
||||
Get status of multiple processing tasks.
|
||||
"""
|
||||
task_id_list = [int(id.strip()) for id in task_ids.split(",")]
|
||||
|
||||
kb = db.query(KnowledgeBase).filter(
|
||||
KnowledgeBase.id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if not kb:
|
||||
raise HTTPException(status_code=404, detail="Knowledge base not found")
|
||||
|
||||
tasks = (
|
||||
db.query(ProcessingTask)
|
||||
.options(
|
||||
selectinload(ProcessingTask.document_upload)
|
||||
)
|
||||
.filter(
|
||||
ProcessingTask.id.in_(task_id_list),
|
||||
ProcessingTask.knowledge_base_id == kb_id
|
||||
)
|
||||
.all()
|
||||
)
|
||||
|
||||
return {
|
||||
task.id: {
|
||||
"document_id": task.document_id,
|
||||
"status": task.status,
|
||||
"error_message": task.error_message,
|
||||
"upload_id": task.document_upload_id,
|
||||
"file_name": task.document_upload.file_name if task.document_upload else None
|
||||
}
|
||||
for task in tasks
|
||||
}
|
||||
|
||||
@router.get("/{kb_id}/documents/{doc_id}", response_model=DocumentResponse)
|
||||
async def get_document(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
kb_id: int,
|
||||
doc_id: int,
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
"""
|
||||
Get document details by ID.
|
||||
"""
|
||||
document = (
|
||||
db.query(Document)
|
||||
.join(KnowledgeBase)
|
||||
.filter(
|
||||
Document.id == doc_id,
|
||||
Document.knowledge_base_id == kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
)
|
||||
.first()
|
||||
)
|
||||
|
||||
if not document:
|
||||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
|
||||
return document
|
||||
|
||||
@router.post("/test-retrieval")
|
||||
async def test_retrieval(
|
||||
request: TestRetrievalRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user)
|
||||
) -> Any:
|
||||
"""
|
||||
Test retrieval quality for a given query against a knowledge base.
|
||||
"""
|
||||
try:
|
||||
kb = db.query(KnowledgeBase).filter(
|
||||
KnowledgeBase.id == request.kb_id,
|
||||
KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if not kb:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Knowledge base {request.kb_id} not found",
|
||||
)
|
||||
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
|
||||
vector_store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{request.kb_id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
|
||||
results = vector_store.similarity_search_with_score(request.query, k=request.top_k)
|
||||
|
||||
response = []
|
||||
for doc, score in results:
|
||||
response.append({
|
||||
"content": doc.page_content,
|
||||
"metadata": doc.metadata,
|
||||
"score": float(score)
|
||||
})
|
||||
|
||||
return {"results": response}
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
84
rag-web-ui/backend/app/api/api_v1/testing.py
Normal file
84
rag-web-ui/backend/app/api/api_v1/testing.py
Normal file
@@ -0,0 +1,84 @@
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.knowledge import Document, KnowledgeBase
|
||||
from app.models.user import User
|
||||
from app.schemas.testing import TestingPipelineRequest, TestingPipelineResponse
|
||||
from app.services.embedding.embedding_factory import EmbeddingsFactory
|
||||
from app.services.retrieval.multi_kb_retriever import MultiKBRetriever, format_retrieval_context
|
||||
from app.services.testing_pipeline import run_testing_pipeline
|
||||
from app.services.vector_store import VectorStoreFactory
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
async def _build_kb_vector_stores(db: Session, knowledge_bases: List[KnowledgeBase]) -> List[Dict[str, Any]]:
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
kb_vector_stores: List[Dict[str, Any]] = []
|
||||
|
||||
for kb in knowledge_bases:
|
||||
documents = db.query(Document).filter(Document.knowledge_base_id == kb.id).all()
|
||||
if not documents:
|
||||
continue
|
||||
|
||||
store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{kb.id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
kb_vector_stores.append({"kb_id": kb.id, "store": store})
|
||||
|
||||
return kb_vector_stores
|
||||
|
||||
|
||||
@router.post("/generate", response_model=TestingPipelineResponse)
|
||||
async def generate_testing_content(
|
||||
*,
|
||||
payload: TestingPipelineRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: Session = Depends(get_db),
|
||||
) -> Any:
|
||||
_ = current_user
|
||||
|
||||
knowledge_context = (payload.knowledge_context or "").strip()
|
||||
if payload.knowledge_base_ids:
|
||||
knowledge_bases = (
|
||||
db.query(KnowledgeBase)
|
||||
.filter(
|
||||
KnowledgeBase.id.in_(payload.knowledge_base_ids),
|
||||
KnowledgeBase.user_id == current_user.id,
|
||||
)
|
||||
.all()
|
||||
)
|
||||
|
||||
kb_vector_stores = await _build_kb_vector_stores(db, knowledge_bases)
|
||||
if kb_vector_stores:
|
||||
retriever = MultiKBRetriever(
|
||||
reranker_weight=settings.RERANKER_WEIGHT,
|
||||
)
|
||||
retrieval_rows = await retriever.retrieve(
|
||||
query=payload.requirement_text,
|
||||
kb_vector_stores=kb_vector_stores,
|
||||
fetch_k_per_kb=max(12, payload.retrieval_top_k * 2),
|
||||
top_k=payload.retrieval_top_k,
|
||||
)
|
||||
if retrieval_rows:
|
||||
knowledge_context = format_retrieval_context(retrieval_rows)
|
||||
|
||||
result = run_testing_pipeline(
|
||||
user_requirement_text=payload.requirement_text,
|
||||
requirement_type_input=payload.requirement_type,
|
||||
debug=payload.debug,
|
||||
knowledge_context=knowledge_context,
|
||||
use_model_generation=payload.use_model_generation,
|
||||
max_items_per_group=payload.max_items_per_group,
|
||||
cases_per_item=payload.cases_per_item,
|
||||
max_focus_points=payload.max_focus_points,
|
||||
max_llm_calls=payload.max_llm_calls,
|
||||
)
|
||||
return result
|
||||
175
rag-web-ui/backend/app/api/api_v1/tools.py
Normal file
175
rag-web-ui/backend/app/api/api_v1/tools.py
Normal file
@@ -0,0 +1,175 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, List
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, File, HTTPException, UploadFile
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.tooling import SRSExtraction, ToolJob
|
||||
from app.models.user import User
|
||||
from app.schemas.tooling import (
|
||||
SRSToolCreateJobResponse,
|
||||
SRSToolJobStatusResponse,
|
||||
SRSToolRequirementsSaveRequest,
|
||||
SRSToolResultResponse,
|
||||
ToolDefinitionResponse,
|
||||
)
|
||||
from app.services.srs_job_service import (
|
||||
build_result_response,
|
||||
ensure_upload_path,
|
||||
replace_requirements,
|
||||
run_srs_job,
|
||||
)
|
||||
from app.tools.registry import ToolRegistry
|
||||
from app.tools.srs_reqs_qwen import get_srs_tool
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
# Register SRS tool when the router is imported.
|
||||
get_srs_tool()
|
||||
|
||||
ALLOWED_EXTENSIONS = {".pdf", ".docx"}
|
||||
|
||||
|
||||
@router.get("", response_model=List[ToolDefinitionResponse])
|
||||
async def list_tools(
|
||||
current_user: User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
_ = current_user
|
||||
return ToolRegistry.list()
|
||||
|
||||
|
||||
@router.post("/srs/jobs", response_model=SRSToolCreateJobResponse)
|
||||
async def create_srs_job(
|
||||
background_tasks: BackgroundTasks,
|
||||
file: UploadFile = File(...),
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
safe_name = Path(file.filename or "").name
|
||||
ext = Path(safe_name).suffix.lower()
|
||||
if ext not in ALLOWED_EXTENSIONS:
|
||||
raise HTTPException(status_code=400, detail="仅支持 .pdf/.docx 文件")
|
||||
|
||||
job = ToolJob(
|
||||
user_id=current_user.id,
|
||||
tool_name="srs.requirement_extractor",
|
||||
status="pending",
|
||||
input_file_name=safe_name,
|
||||
input_file_path="",
|
||||
)
|
||||
db.add(job)
|
||||
db.commit()
|
||||
db.refresh(job)
|
||||
|
||||
target_path = ensure_upload_path(job.id, safe_name)
|
||||
try:
|
||||
content = await file.read()
|
||||
target_path.write_bytes(content)
|
||||
except Exception as exc:
|
||||
job.status = "failed"
|
||||
job.error_message = f"保存上传文件失败: {exc}"
|
||||
db.add(job)
|
||||
db.commit()
|
||||
raise HTTPException(status_code=500, detail="上传文件保存失败")
|
||||
|
||||
job.input_file_path = str(target_path.resolve())
|
||||
db.add(job)
|
||||
db.commit()
|
||||
|
||||
background_tasks.add_task(run_srs_job, job.id)
|
||||
|
||||
return {
|
||||
"job_id": job.id,
|
||||
"status": job.status,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/srs/jobs/{job_id}", response_model=SRSToolJobStatusResponse)
|
||||
async def get_srs_job_status(
|
||||
job_id: int,
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
job = (
|
||||
db.query(ToolJob)
|
||||
.filter(ToolJob.id == job_id, ToolJob.user_id == current_user.id)
|
||||
.first()
|
||||
)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="任务不存在")
|
||||
|
||||
extraction = (
|
||||
db.query(SRSExtraction)
|
||||
.filter(SRSExtraction.job_id == job.id)
|
||||
.first()
|
||||
)
|
||||
|
||||
return {
|
||||
"job_id": job.id,
|
||||
"tool_name": job.tool_name,
|
||||
"status": job.status,
|
||||
"error_message": job.error_message,
|
||||
"extraction_id": extraction.id if extraction else None,
|
||||
"started_at": job.started_at,
|
||||
"completed_at": job.completed_at,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/srs/jobs/{job_id}/result", response_model=SRSToolResultResponse)
|
||||
async def get_srs_job_result(
|
||||
job_id: int,
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
job = (
|
||||
db.query(ToolJob)
|
||||
.filter(ToolJob.id == job_id, ToolJob.user_id == current_user.id)
|
||||
.first()
|
||||
)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="任务不存在")
|
||||
if job.status != "completed":
|
||||
raise HTTPException(status_code=409, detail="任务尚未完成")
|
||||
|
||||
extraction = (
|
||||
db.query(SRSExtraction)
|
||||
.filter(SRSExtraction.job_id == job.id)
|
||||
.first()
|
||||
)
|
||||
if not extraction:
|
||||
raise HTTPException(status_code=404, detail="任务结果不存在")
|
||||
|
||||
return build_result_response(job, extraction)
|
||||
|
||||
|
||||
@router.put("/srs/jobs/{job_id}/requirements", response_model=SRSToolResultResponse)
|
||||
async def save_srs_requirements(
|
||||
job_id: int,
|
||||
payload: SRSToolRequirementsSaveRequest,
|
||||
db: Session = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
) -> Any:
|
||||
job = (
|
||||
db.query(ToolJob)
|
||||
.filter(ToolJob.id == job_id, ToolJob.user_id == current_user.id)
|
||||
.first()
|
||||
)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="任务不存在")
|
||||
|
||||
extraction = (
|
||||
db.query(SRSExtraction)
|
||||
.filter(SRSExtraction.job_id == job.id)
|
||||
.first()
|
||||
)
|
||||
if not extraction:
|
||||
raise HTTPException(status_code=404, detail="任务结果不存在")
|
||||
|
||||
replace_requirements(db, extraction, [item.dict() for item in payload.requirements])
|
||||
db.add(extraction)
|
||||
db.commit()
|
||||
db.refresh(extraction)
|
||||
|
||||
return build_result_response(job, extraction)
|
||||
6
rag-web-ui/backend/app/api/openapi/api.py
Normal file
6
rag-web-ui/backend/app/api/openapi/api.py
Normal file
@@ -0,0 +1,6 @@
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.api.openapi import knowledge
|
||||
|
||||
router = APIRouter()
|
||||
router.include_router(knowledge.router, prefix="/knowledge", tags=["knowledge"])
|
||||
60
rag-web-ui/backend/app/api/openapi/knowledge.py
Normal file
60
rag-web-ui/backend/app/api/openapi/knowledge.py
Normal file
@@ -0,0 +1,60 @@
|
||||
from typing import Any, List
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
from langchain_chroma import Chroma
|
||||
from app.services.vector_store import VectorStoreFactory
|
||||
|
||||
from app import models
|
||||
from app.db.session import get_db
|
||||
from app.core.security import get_api_key_user
|
||||
from app.core.config import settings
|
||||
from app.services.embedding.embedding_factory import EmbeddingsFactory
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.get("/{knowledge_base_id}/query")
|
||||
def query_knowledge_base(
|
||||
*,
|
||||
db: Session = Depends(get_db),
|
||||
knowledge_base_id: int,
|
||||
query: str,
|
||||
top_k: int = 3,
|
||||
current_user: models.User = Depends(get_api_key_user),
|
||||
) -> Any:
|
||||
"""
|
||||
Query a specific knowledge base using API key authentication
|
||||
"""
|
||||
try:
|
||||
kb = db.query(models.KnowledgeBase).filter(
|
||||
models.KnowledgeBase.id == knowledge_base_id,
|
||||
models.KnowledgeBase.user_id == current_user.id
|
||||
).first()
|
||||
|
||||
if not kb:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Knowledge base {knowledge_base_id} not found",
|
||||
)
|
||||
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
|
||||
vector_store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{knowledge_base_id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
|
||||
results = vector_store.similarity_search_with_score(query, k=top_k)
|
||||
|
||||
response = []
|
||||
for doc, score in results:
|
||||
response.append({
|
||||
"content": doc.page_content,
|
||||
"metadata": doc.metadata,
|
||||
"score": float(score)
|
||||
})
|
||||
|
||||
return {"results": response}
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
0
rag-web-ui/backend/app/core/__init__.py
Normal file
0
rag-web-ui/backend/app/core/__init__.py
Normal file
123
rag-web-ui/backend/app/core/config.py
Normal file
123
rag-web-ui/backend/app/core/config.py
Normal file
@@ -0,0 +1,123 @@
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
PROJECT_NAME: str = "RAG Web UI" # Project name
|
||||
VERSION: str = "0.1.0" # Project version
|
||||
API_V1_STR: str = "/api" # API version string
|
||||
|
||||
# MySQL settings
|
||||
MYSQL_SERVER: str = os.getenv("MYSQL_SERVER", "localhost")
|
||||
MYSQL_PORT: int = int(os.getenv("MYSQL_PORT", "3306"))
|
||||
MYSQL_USER: str = os.getenv("MYSQL_USER", "ragagent")
|
||||
MYSQL_PASSWORD: str = os.getenv("MYSQL_PASSWORD", "ragagent")
|
||||
MYSQL_DATABASE: str = os.getenv("MYSQL_DATABASE", "ragagent")
|
||||
SQLALCHEMY_DATABASE_URI: Optional[str] = None
|
||||
|
||||
@property
|
||||
def get_database_url(self) -> str:
|
||||
if self.SQLALCHEMY_DATABASE_URI:
|
||||
return self.SQLALCHEMY_DATABASE_URI
|
||||
return (
|
||||
f"mysql+mysqlconnector://{self.MYSQL_USER}:{self.MYSQL_PASSWORD}"
|
||||
f"@{self.MYSQL_SERVER}:{self.MYSQL_PORT}/{self.MYSQL_DATABASE}"
|
||||
)
|
||||
|
||||
# JWT settings
|
||||
SECRET_KEY: str = os.getenv("SECRET_KEY", "your-secret-key-here")
|
||||
ALGORITHM: str = "HS256"
|
||||
ACCESS_TOKEN_EXPIRE_MINUTES: int = int(os.getenv("ACCESS_TOKEN_EXPIRE_MINUTES", "10080"))
|
||||
|
||||
# Chat Provider settings
|
||||
CHAT_PROVIDER: str = os.getenv("CHAT_PROVIDER", "openai")
|
||||
|
||||
# Embeddings settings
|
||||
EMBEDDINGS_PROVIDER: str = os.getenv("EMBEDDINGS_PROVIDER", "openai")
|
||||
|
||||
# MinIO settings
|
||||
MINIO_ENDPOINT: str = os.getenv("MINIO_ENDPOINT", "localhost:9000")
|
||||
MINIO_ACCESS_KEY: str = os.getenv("MINIO_ACCESS_KEY", "minioadmin")
|
||||
MINIO_SECRET_KEY: str = os.getenv("MINIO_SECRET_KEY", "minioadmin")
|
||||
MINIO_BUCKET_NAME: str = os.getenv("MINIO_BUCKET_NAME", "documents")
|
||||
|
||||
# Shared model API key fallback
|
||||
API_KEY: str = os.getenv("API_KEY", "")
|
||||
|
||||
# OpenAI settings
|
||||
OPENAI_API_BASE: str = os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1")
|
||||
OPENAI_API_KEY: str = os.getenv(
|
||||
"OPENAI_API_KEY", os.getenv("API_KEY", "your-openai-api-key-here")
|
||||
)
|
||||
OPENAI_MODEL: str = os.getenv("OPENAI_MODEL", "gpt-4")
|
||||
OPENAI_EMBEDDINGS_MODEL: str = os.getenv("OPENAI_EMBEDDINGS_MODEL", "text-embedding-ada-002")
|
||||
|
||||
# DashScope settings
|
||||
DASH_SCOPE_API_KEY: str = os.getenv(
|
||||
"DASH_SCOPE_API_KEY",
|
||||
os.getenv("DASHSCOPE_API_KEY", os.getenv("API_KEY", "")),
|
||||
)
|
||||
DASH_SCOPE_API_BASE: str = os.getenv(
|
||||
"DASH_SCOPE_API_BASE", "https://dashscope.aliyuncs.com/compatible-mode/v1"
|
||||
)
|
||||
DASH_SCOPE_CHAT_MODEL: str = os.getenv("DASH_SCOPE_CHAT_MODEL", "qwen3-max")
|
||||
DASH_SCOPE_EMBEDDINGS_MODEL: str = os.getenv("DASH_SCOPE_EMBEDDINGS_MODEL", "")
|
||||
|
||||
# Vector Store settings
|
||||
VECTOR_STORE_TYPE: str = os.getenv("VECTOR_STORE_TYPE", "chroma")
|
||||
|
||||
# External reranker settings
|
||||
RERANKER_API_URL: str = os.getenv("RERANKER_API_URL", "")
|
||||
RERANKER_API_KEY: str = os.getenv(
|
||||
"RERANKER_API_KEY",
|
||||
os.getenv(
|
||||
"DASH_SCOPE_API_KEY",
|
||||
os.getenv("DASHSCOPE_API_KEY", os.getenv("API_KEY", "")),
|
||||
),
|
||||
)
|
||||
RERANKER_MODEL: str = os.getenv("RERANKER_MODEL", "")
|
||||
RERANKER_TIMEOUT_SECONDS: float = float(os.getenv("RERANKER_TIMEOUT_SECONDS", "8"))
|
||||
RERANKER_WEIGHT: float = float(os.getenv("RERANKER_WEIGHT", "0.75"))
|
||||
|
||||
# GraphRAG settings
|
||||
GRAPHRAG_ENABLED: bool = os.getenv("GRAPHRAG_ENABLED", "false").lower() == "true"
|
||||
GRAPHRAG_WORKING_DIR: str = os.getenv("GRAPHRAG_WORKING_DIR", "./graphrag_cache")
|
||||
GRAPHRAG_GRAPH_STORAGE: str = os.getenv("GRAPHRAG_GRAPH_STORAGE", "neo4j")
|
||||
GRAPHRAG_QUERY_LEVEL: int = int(os.getenv("GRAPHRAG_QUERY_LEVEL", "2"))
|
||||
GRAPHRAG_LOCAL_TOP_K: int = int(os.getenv("GRAPHRAG_LOCAL_TOP_K", "20"))
|
||||
GRAPHRAG_ENTITY_EXTRACT_MAX_GLEANING: int = int(os.getenv("GRAPHRAG_ENTITY_EXTRACT_MAX_GLEANING", "1"))
|
||||
GRAPHRAG_EMBEDDING_DIM: int = int(os.getenv("GRAPHRAG_EMBEDDING_DIM", "1024"))
|
||||
GRAPHRAG_EMBEDDING_MAX_TOKEN_SIZE: int = int(os.getenv("GRAPHRAG_EMBEDDING_MAX_TOKEN_SIZE", "8192"))
|
||||
|
||||
# Neo4j settings
|
||||
NEO4J_URL: str = os.getenv("NEO4J_URL", "bolt://localhost:7687")
|
||||
NEO4J_USERNAME: str = os.getenv("NEO4J_USERNAME", "neo4j")
|
||||
NEO4J_PASSWORD: str = os.getenv("NEO4J_PASSWORD", "neo4j")
|
||||
|
||||
# Chroma DB settings
|
||||
CHROMA_DB_HOST: str = os.getenv("CHROMA_DB_HOST", "chromadb")
|
||||
CHROMA_DB_PORT: int = int(os.getenv("CHROMA_DB_PORT", "8000"))
|
||||
|
||||
# Qdrant DB settings
|
||||
QDRANT_URL: str = os.getenv("QDRANT_URL", "http://localhost:6333")
|
||||
QDRANT_PREFER_GRPC: bool = os.getenv("QDRANT_PREFER_GRPC", "true").lower() == "true"
|
||||
|
||||
# Deepseek settings
|
||||
DEEPSEEK_API_KEY: str = ""
|
||||
DEEPSEEK_API_BASE: str = "https://api.deepseek.com/v1" # 默认 API 地址
|
||||
DEEPSEEK_MODEL: str = "deepseek-chat" # 默认模型名称
|
||||
|
||||
# Ollama settings
|
||||
OLLAMA_API_BASE: str = "http://localhost:11434"
|
||||
OLLAMA_MODEL: str = "deepseek-r1:7b"
|
||||
OLLAMA_EMBEDDINGS_MODEL: str = os.getenv(
|
||||
"OLLAMA_EMBEDDINGS_MODEL", "nomic-embed-text"
|
||||
) # Added this line
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
|
||||
|
||||
settings = Settings()
|
||||
29
rag-web-ui/backend/app/core/minio.py
Normal file
29
rag-web-ui/backend/app/core/minio.py
Normal file
@@ -0,0 +1,29 @@
|
||||
import logging
|
||||
from minio import Minio
|
||||
from app.core.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def get_minio_client() -> Minio:
|
||||
"""
|
||||
Get a MinIO client instance.
|
||||
"""
|
||||
logger.info("Creating MinIO client instance.")
|
||||
return Minio(
|
||||
settings.MINIO_ENDPOINT,
|
||||
access_key=settings.MINIO_ACCESS_KEY,
|
||||
secret_key=settings.MINIO_SECRET_KEY,
|
||||
secure=False # Set to True if using HTTPS
|
||||
)
|
||||
|
||||
def init_minio():
|
||||
"""
|
||||
Initialize MinIO by creating the bucket if it doesn't exist.
|
||||
"""
|
||||
client = get_minio_client()
|
||||
logger.info(f"Checking if bucket {settings.MINIO_BUCKET_NAME} exists.")
|
||||
if not client.bucket_exists(settings.MINIO_BUCKET_NAME):
|
||||
logger.info(f"Bucket {settings.MINIO_BUCKET_NAME} does not exist. Creating bucket.")
|
||||
client.make_bucket(settings.MINIO_BUCKET_NAME)
|
||||
else:
|
||||
logger.info(f"Bucket {settings.MINIO_BUCKET_NAME} already exists.")
|
||||
27
rag-web-ui/backend/app/core/runtime_checks.py
Normal file
27
rag-web-ui/backend/app/core/runtime_checks.py
Normal file
@@ -0,0 +1,27 @@
|
||||
import logging
|
||||
|
||||
from app.core.config import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def validate_runtime_settings(settings: Settings) -> None:
|
||||
errors = []
|
||||
|
||||
if settings.GRAPHRAG_ENABLED:
|
||||
if settings.GRAPHRAG_GRAPH_STORAGE.lower() not in {"neo4j", "networkx"}:
|
||||
errors.append("GRAPHRAG_GRAPH_STORAGE must be either 'neo4j' or 'networkx'.")
|
||||
|
||||
if settings.GRAPHRAG_GRAPH_STORAGE.lower() == "neo4j":
|
||||
if not settings.NEO4J_URL:
|
||||
errors.append("NEO4J_URL is required when GraphRAG Neo4j storage is enabled.")
|
||||
if not settings.NEO4J_USERNAME:
|
||||
errors.append("NEO4J_USERNAME is required when GraphRAG Neo4j storage is enabled.")
|
||||
if not settings.NEO4J_PASSWORD:
|
||||
errors.append("NEO4J_PASSWORD is required when GraphRAG Neo4j storage is enabled.")
|
||||
|
||||
if settings.RERANKER_API_URL and not settings.RERANKER_MODEL:
|
||||
logger.warning("RERANKER_API_URL is configured but RERANKER_MODEL is empty. The API may reject requests.")
|
||||
|
||||
if errors:
|
||||
raise ValueError("Runtime configuration validation failed: " + " | ".join(errors))
|
||||
84
rag-web-ui/backend/app/core/security.py
Normal file
84
rag-web-ui/backend/app/core/security.py
Normal file
@@ -0,0 +1,84 @@
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional
|
||||
from jose import JWTError, jwt
|
||||
import bcrypt
|
||||
from app.core.config import settings
|
||||
from fastapi import Depends, HTTPException, status, Security
|
||||
from fastapi.security import OAuth2PasswordBearer, APIKeyHeader
|
||||
from sqlalchemy.orm import Session
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.services.api_key import APIKeyService
|
||||
|
||||
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="/api/v1/auth/login/access-token")
|
||||
api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
return bcrypt.checkpw(plain_password.encode("utf-8"), hashed_password.encode("utf-8"))
|
||||
|
||||
def get_password_hash(password: str) -> str:
|
||||
return bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt()).decode("utf-8")
|
||||
|
||||
def create_access_token(data: dict, expires_delta: Optional[timedelta] = None) -> str:
|
||||
to_encode = data.copy()
|
||||
if expires_delta:
|
||||
expire = datetime.utcnow() + expires_delta
|
||||
else:
|
||||
expire = datetime.utcnow() + timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||
to_encode.update({"exp": expire})
|
||||
encoded_jwt = jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
|
||||
return encoded_jwt
|
||||
|
||||
def get_current_user(
|
||||
db: Session = Depends(get_db),
|
||||
token: str = Depends(oauth2_scheme)
|
||||
) -> User:
|
||||
credentials_exception = HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Could not validate credentials",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
try:
|
||||
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
|
||||
username: str = payload.get("sub")
|
||||
if username is None:
|
||||
raise credentials_exception
|
||||
except JWTError:
|
||||
raise credentials_exception
|
||||
|
||||
user = db.query(User).filter(User.username == username).first()
|
||||
if user is None:
|
||||
raise credentials_exception
|
||||
if not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Inactive user",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
return user
|
||||
|
||||
def get_api_key_user(
|
||||
db: Session = Depends(get_db),
|
||||
api_key: str = Security(api_key_header),
|
||||
) -> User:
|
||||
if not api_key:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="API key header missing",
|
||||
)
|
||||
|
||||
api_key_obj = APIKeyService.get_api_key_by_key(db=db, key=api_key)
|
||||
if not api_key_obj:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid API key",
|
||||
)
|
||||
|
||||
if not api_key_obj.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Inactive API key",
|
||||
)
|
||||
|
||||
APIKeyService.update_last_used(db=db, api_key=api_key_obj)
|
||||
return api_key_obj.user
|
||||
0
rag-web-ui/backend/app/db/__init__.py
Normal file
0
rag-web-ui/backend/app/db/__init__.py
Normal file
13
rag-web-ui/backend/app/db/session.py
Normal file
13
rag-web-ui/backend/app/db/session.py
Normal file
@@ -0,0 +1,13 @@
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from app.core.config import settings
|
||||
|
||||
engine = create_engine(settings.get_database_url)
|
||||
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
||||
|
||||
def get_db():
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
47
rag-web-ui/backend/app/main.py
Normal file
47
rag-web-ui/backend/app/main.py
Normal file
@@ -0,0 +1,47 @@
|
||||
import logging
|
||||
|
||||
from app.api.api_v1.api import api_router
|
||||
from app.api.openapi.api import router as openapi_router
|
||||
from app.core.config import settings
|
||||
from app.core.minio import init_minio
|
||||
from app.core.runtime_checks import validate_runtime_settings
|
||||
from app.startup.migarate import DatabaseMigrator
|
||||
from fastapi import FastAPI
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
)
|
||||
|
||||
app = FastAPI(
|
||||
title=settings.PROJECT_NAME,
|
||||
version=settings.VERSION,
|
||||
openapi_url=f"{settings.API_V1_STR}/openapi.json",
|
||||
)
|
||||
|
||||
# Include routers
|
||||
app.include_router(api_router, prefix=settings.API_V1_STR)
|
||||
app.include_router(openapi_router, prefix="/openapi")
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup_event():
|
||||
validate_runtime_settings(settings)
|
||||
# Initialize MinIO
|
||||
init_minio()
|
||||
# Run database migrations
|
||||
migrator = DatabaseMigrator(settings.get_database_url)
|
||||
migrator.run_migrations()
|
||||
|
||||
|
||||
@app.get("/")
|
||||
def root():
|
||||
return {"message": "Welcome to RAG Web UI API"}
|
||||
|
||||
|
||||
@app.get("/api/health")
|
||||
async def health_check():
|
||||
return {
|
||||
"status": "healthy",
|
||||
"version": settings.VERSION,
|
||||
}
|
||||
18
rag-web-ui/backend/app/models/__init__.py
Normal file
18
rag-web-ui/backend/app/models/__init__.py
Normal file
@@ -0,0 +1,18 @@
|
||||
from .user import User
|
||||
from .knowledge import KnowledgeBase, Document, DocumentChunk
|
||||
from .chat import Chat, Message
|
||||
from .api_key import APIKey
|
||||
from .tooling import ToolJob, SRSExtraction, SRSRequirement
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
"KnowledgeBase",
|
||||
"Document",
|
||||
"DocumentChunk",
|
||||
"Chat",
|
||||
"Message",
|
||||
"APIKey",
|
||||
"ToolJob",
|
||||
"SRSExtraction",
|
||||
"SRSRequirement",
|
||||
]
|
||||
18
rag-web-ui/backend/app/models/api_key.py
Normal file
18
rag-web-ui/backend/app/models/api_key.py
Normal file
@@ -0,0 +1,18 @@
|
||||
from sqlalchemy import Column, Integer, String, Boolean, DateTime, ForeignKey, VARCHAR
|
||||
from sqlalchemy.orm import relationship
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.models.base import Base, TimestampMixin
|
||||
|
||||
class APIKey(Base, TimestampMixin):
|
||||
__tablename__ = "api_keys"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
key = Column(VARCHAR(128), unique=True, index=True, nullable=False)
|
||||
name = Column(String(255), nullable=False)
|
||||
user_id = Column(Integer, ForeignKey("users.id"), nullable=False)
|
||||
is_active = Column(Boolean, default=True, nullable=False)
|
||||
last_used_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
# Relationships
|
||||
user = relationship("User", back_populates="api_keys")
|
||||
9
rag-web-ui/backend/app/models/base.py
Normal file
9
rag-web-ui/backend/app/models/base.py
Normal file
@@ -0,0 +1,9 @@
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy import Column, Integer, DateTime
|
||||
from datetime import datetime
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
class TimestampMixin:
|
||||
created_at = Column(DateTime, default=datetime.utcnow, nullable=False)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow, nullable=False)
|
||||
39
rag-web-ui/backend/app/models/chat.py
Normal file
39
rag-web-ui/backend/app/models/chat.py
Normal file
@@ -0,0 +1,39 @@
|
||||
from sqlalchemy import Column, Integer, String, ForeignKey, Boolean, Table
|
||||
from sqlalchemy.dialects.mysql import LONGTEXT
|
||||
from sqlalchemy.orm import relationship
|
||||
from app.models.base import Base, TimestampMixin
|
||||
|
||||
# Association table for many-to-many relationship between Chat and KnowledgeBase
|
||||
chat_knowledge_bases = Table(
|
||||
"chat_knowledge_bases",
|
||||
Base.metadata,
|
||||
Column("chat_id", Integer, ForeignKey("chats.id"), primary_key=True),
|
||||
Column("knowledge_base_id", Integer, ForeignKey("knowledge_bases.id"), primary_key=True),
|
||||
)
|
||||
|
||||
class Chat(Base, TimestampMixin):
|
||||
__tablename__ = "chats"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
title = Column(String(255), nullable=False)
|
||||
user_id = Column(Integer, ForeignKey("users.id"), nullable=False)
|
||||
|
||||
# Relationships
|
||||
messages = relationship("Message", back_populates="chat", cascade="all, delete-orphan")
|
||||
user = relationship("User", back_populates="chats")
|
||||
knowledge_bases = relationship(
|
||||
"KnowledgeBase",
|
||||
secondary=chat_knowledge_bases,
|
||||
backref="chats"
|
||||
)
|
||||
|
||||
class Message(Base, TimestampMixin):
|
||||
__tablename__ = "messages"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
content = Column(LONGTEXT, nullable=False)
|
||||
role = Column(String(50), nullable=False)
|
||||
chat_id = Column(Integer, ForeignKey("chats.id"), nullable=False)
|
||||
|
||||
# Relationships
|
||||
chat = relationship("Chat", back_populates="messages")
|
||||
97
rag-web-ui/backend/app/models/knowledge.py
Normal file
97
rag-web-ui/backend/app/models/knowledge.py
Normal file
@@ -0,0 +1,97 @@
|
||||
from sqlalchemy import Column, Integer, String, ForeignKey, Text, DateTime, JSON, BigInteger, TIMESTAMP, text
|
||||
from sqlalchemy.dialects.mysql import LONGTEXT
|
||||
from sqlalchemy.orm import relationship
|
||||
from app.models.base import Base, TimestampMixin
|
||||
from datetime import datetime
|
||||
import sqlalchemy as sa
|
||||
|
||||
class KnowledgeBase(Base, TimestampMixin):
|
||||
__tablename__ = "knowledge_bases"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
name = Column(String(255), nullable=False)
|
||||
description = Column(LONGTEXT)
|
||||
user_id = Column(Integer, ForeignKey("users.id"), nullable=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
# Relationships
|
||||
documents = relationship("Document", back_populates="knowledge_base", cascade="all, delete-orphan")
|
||||
user = relationship("User", back_populates="knowledge_bases")
|
||||
processing_tasks = relationship("ProcessingTask", back_populates="knowledge_base")
|
||||
chunks = relationship("DocumentChunk", back_populates="knowledge_base", cascade="all, delete-orphan")
|
||||
document_uploads = relationship("DocumentUpload", back_populates="knowledge_base", cascade="all, delete-orphan")
|
||||
|
||||
class Document(Base, TimestampMixin):
|
||||
__tablename__ = "documents"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
file_path = Column(String(255), nullable=False) # Path in MinIO
|
||||
file_name = Column(String(255), nullable=False) # Actual file name
|
||||
file_size = Column(BigInteger, nullable=False) # File size in bytes
|
||||
content_type = Column(String(100), nullable=False) # MIME type
|
||||
file_hash = Column(String(64), index=True) # SHA-256 hash of file content
|
||||
knowledge_base_id = Column(Integer, ForeignKey("knowledge_bases.id"), nullable=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
# Relationships
|
||||
knowledge_base = relationship("KnowledgeBase", back_populates="documents")
|
||||
processing_tasks = relationship("ProcessingTask", back_populates="document")
|
||||
chunks = relationship("DocumentChunk", back_populates="document", cascade="all, delete-orphan")
|
||||
|
||||
__table_args__ = (
|
||||
# Ensure file_name is unique within each knowledge base
|
||||
sa.UniqueConstraint('knowledge_base_id', 'file_name', name='uq_kb_file_name'),
|
||||
)
|
||||
|
||||
class DocumentUpload(Base):
|
||||
__tablename__ = "document_uploads"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
knowledge_base_id = Column(Integer, ForeignKey("knowledge_bases.id", ondelete="CASCADE"), nullable=False)
|
||||
file_name = Column(String, nullable=False)
|
||||
file_hash = Column(String, nullable=False)
|
||||
file_size = Column(BigInteger, nullable=False)
|
||||
content_type = Column(String, nullable=False)
|
||||
temp_path = Column(String, nullable=False)
|
||||
created_at = Column(TIMESTAMP, nullable=False, server_default=text("now()"))
|
||||
status = Column(String, nullable=False, server_default="pending")
|
||||
error_message = Column(Text)
|
||||
|
||||
# Relationships
|
||||
knowledge_base = relationship("KnowledgeBase", back_populates="document_uploads")
|
||||
|
||||
class ProcessingTask(Base):
|
||||
__tablename__ = "processing_tasks"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
knowledge_base_id = Column(Integer, ForeignKey("knowledge_bases.id"))
|
||||
document_id = Column(Integer, ForeignKey("documents.id"), nullable=True)
|
||||
document_upload_id = Column(Integer, ForeignKey("document_uploads.id"), nullable=True)
|
||||
status = Column(String(50), default="pending") # pending, processing, completed, failed
|
||||
error_message = Column(Text, nullable=True)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
knowledge_base = relationship("KnowledgeBase", back_populates="processing_tasks")
|
||||
document = relationship("Document", back_populates="processing_tasks")
|
||||
document_upload = relationship("DocumentUpload", backref="processing_tasks")
|
||||
|
||||
class DocumentChunk(Base, TimestampMixin):
|
||||
__tablename__ = "document_chunks"
|
||||
|
||||
id = Column(String(64), primary_key=True) # SHA-256 hash as ID
|
||||
kb_id = Column(Integer, ForeignKey("knowledge_bases.id"), nullable=False)
|
||||
document_id = Column(Integer, ForeignKey("documents.id"), nullable=False)
|
||||
file_name = Column(String(255), nullable=False)
|
||||
chunk_metadata = Column(JSON, nullable=True)
|
||||
hash = Column(String(64), nullable=False, index=True) # Content hash for change detection
|
||||
|
||||
# Relationships
|
||||
knowledge_base = relationship("KnowledgeBase", back_populates="chunks")
|
||||
document = relationship("Document", back_populates="chunks")
|
||||
|
||||
__table_args__ = (
|
||||
sa.Index('idx_kb_file_name', 'kb_id', 'file_name'),
|
||||
)
|
||||
76
rag-web-ui/backend/app/models/tooling.py
Normal file
76
rag-web-ui/backend/app/models/tooling.py
Normal file
@@ -0,0 +1,76 @@
|
||||
from datetime import datetime
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import Column, DateTime, ForeignKey, Integer, JSON, String, Text
|
||||
from sqlalchemy.dialects.mysql import LONGTEXT
|
||||
from sqlalchemy.orm import relationship
|
||||
|
||||
from app.models.base import Base, TimestampMixin
|
||||
|
||||
|
||||
class ToolJob(Base, TimestampMixin):
|
||||
__tablename__ = "tool_jobs"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id"), nullable=False, index=True)
|
||||
tool_name = Column(String(128), nullable=False, index=True)
|
||||
status = Column(String(32), nullable=False, default="pending")
|
||||
input_file_name = Column(String(255), nullable=False)
|
||||
input_file_path = Column(String(512), nullable=False)
|
||||
error_message = Column(Text, nullable=True)
|
||||
started_at = Column(DateTime, nullable=True)
|
||||
completed_at = Column(DateTime, nullable=True)
|
||||
output_summary = Column(JSON, nullable=True)
|
||||
|
||||
user = relationship("User")
|
||||
srs_extraction = relationship(
|
||||
"SRSExtraction",
|
||||
back_populates="job",
|
||||
uselist=False,
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
|
||||
|
||||
class SRSExtraction(Base, TimestampMixin):
|
||||
__tablename__ = "srs_extractions"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
job_id = Column(Integer, ForeignKey("tool_jobs.id", ondelete="CASCADE"), nullable=False, unique=True)
|
||||
document_name = Column(String(255), nullable=False)
|
||||
document_title = Column(String(255), nullable=False)
|
||||
generated_at = Column(DateTime, default=datetime.utcnow, nullable=False)
|
||||
total_requirements = Column(Integer, nullable=False, default=0)
|
||||
statistics = Column(JSON, nullable=True)
|
||||
raw_output = Column(JSON, nullable=True)
|
||||
|
||||
job = relationship("ToolJob", back_populates="srs_extraction")
|
||||
requirements = relationship(
|
||||
"SRSRequirement",
|
||||
back_populates="extraction",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="SRSRequirement.sort_order",
|
||||
)
|
||||
|
||||
|
||||
class SRSRequirement(Base, TimestampMixin):
|
||||
__tablename__ = "srs_requirements"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
extraction_id = Column(Integer, ForeignKey("srs_extractions.id", ondelete="CASCADE"), nullable=False)
|
||||
requirement_uid = Column(String(64), nullable=False)
|
||||
title = Column(String(255), nullable=False)
|
||||
description = Column(LONGTEXT, nullable=False)
|
||||
priority = Column(String(16), nullable=False, default="中")
|
||||
acceptance_criteria = Column(JSON, nullable=False)
|
||||
source_field = Column(String(255), nullable=False)
|
||||
section_number = Column(String(64), nullable=True)
|
||||
section_title = Column(String(255), nullable=True)
|
||||
requirement_type = Column(String(64), nullable=True)
|
||||
sort_order = Column(Integer, nullable=False, default=0)
|
||||
|
||||
extraction = relationship("SRSExtraction", back_populates="requirements")
|
||||
|
||||
__table_args__ = (
|
||||
sa.UniqueConstraint("extraction_id", "requirement_uid", name="uq_srs_extraction_requirement_uid"),
|
||||
sa.Index("idx_srs_requirements_extraction_sort", "extraction_id", "sort_order"),
|
||||
)
|
||||
18
rag-web-ui/backend/app/models/user.py
Normal file
18
rag-web-ui/backend/app/models/user.py
Normal file
@@ -0,0 +1,18 @@
|
||||
from sqlalchemy import Boolean, Column, Integer, String
|
||||
from sqlalchemy.orm import relationship
|
||||
from app.models.base import Base, TimestampMixin
|
||||
|
||||
class User(Base, TimestampMixin):
|
||||
__tablename__ = "users"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
email = Column(String(255), unique=True, index=True, nullable=False)
|
||||
username = Column(String(255), unique=True, index=True, nullable=False)
|
||||
hashed_password = Column(String(255), nullable=False)
|
||||
is_active = Column(Boolean, default=True)
|
||||
is_superuser = Column(Boolean, default=False)
|
||||
|
||||
# Relationships
|
||||
knowledge_bases = relationship("KnowledgeBase", back_populates="user")
|
||||
chats = relationship("Chat", back_populates="user")
|
||||
api_keys = relationship("APIKey", back_populates="user", cascade="all, delete-orphan")
|
||||
12
rag-web-ui/backend/app/schemas/__init__.py
Normal file
12
rag-web-ui/backend/app/schemas/__init__.py
Normal file
@@ -0,0 +1,12 @@
|
||||
from .api_key import APIKey, APIKeyCreate, APIKeyUpdate, APIKeyInDB
|
||||
from .user import UserBase, UserCreate, UserUpdate, UserResponse
|
||||
from .token import Token, TokenPayload
|
||||
from .knowledge import KnowledgeBaseBase, KnowledgeBaseCreate, KnowledgeBaseUpdate, KnowledgeBaseResponse
|
||||
from .testing import (
|
||||
ExpectedResultEntry,
|
||||
StepLogEntry,
|
||||
TestCaseEntry,
|
||||
TestItemEntry,
|
||||
TestingPipelineRequest,
|
||||
TestingPipelineResponse,
|
||||
)
|
||||
28
rag-web-ui/backend/app/schemas/api_key.py
Normal file
28
rag-web-ui/backend/app/schemas/api_key.py
Normal file
@@ -0,0 +1,28 @@
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from pydantic import BaseModel
|
||||
|
||||
class APIKeyBase(BaseModel):
|
||||
name: str
|
||||
is_active: bool = True
|
||||
|
||||
class APIKeyCreate(APIKeyBase):
|
||||
pass
|
||||
|
||||
class APIKeyUpdate(BaseModel):
|
||||
name: Optional[str] = None
|
||||
is_active: Optional[bool] = None
|
||||
|
||||
class APIKey(APIKeyBase):
|
||||
id: int
|
||||
key: str
|
||||
user_id: int
|
||||
last_used_at: Optional[datetime] = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
class APIKeyInDB(APIKey):
|
||||
pass
|
||||
39
rag-web-ui/backend/app/schemas/chat.py
Normal file
39
rag-web-ui/backend/app/schemas/chat.py
Normal file
@@ -0,0 +1,39 @@
|
||||
from pydantic import BaseModel
|
||||
from typing import List, Optional
|
||||
from datetime import datetime
|
||||
|
||||
class MessageBase(BaseModel):
|
||||
content: str
|
||||
role: str
|
||||
|
||||
class MessageCreate(MessageBase):
|
||||
chat_id: int
|
||||
|
||||
class MessageResponse(MessageBase):
|
||||
id: int
|
||||
chat_id: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
class ChatBase(BaseModel):
|
||||
title: str
|
||||
|
||||
class ChatCreate(ChatBase):
|
||||
knowledge_base_ids: List[int]
|
||||
|
||||
class ChatUpdate(ChatBase):
|
||||
knowledge_base_ids: Optional[List[int]] = None
|
||||
|
||||
class ChatResponse(ChatBase):
|
||||
id: int
|
||||
user_id: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
messages: List[MessageResponse] = []
|
||||
knowledge_base_ids: List[int] = []
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
85
rag-web-ui/backend/app/schemas/knowledge.py
Normal file
85
rag-web-ui/backend/app/schemas/knowledge.py
Normal file
@@ -0,0 +1,85 @@
|
||||
from typing import Optional, List
|
||||
from datetime import datetime
|
||||
from pydantic import BaseModel
|
||||
|
||||
class KnowledgeBaseBase(BaseModel):
|
||||
name: str
|
||||
description: Optional[str] = None
|
||||
|
||||
class KnowledgeBaseCreate(KnowledgeBaseBase):
|
||||
pass
|
||||
|
||||
class KnowledgeBaseUpdate(KnowledgeBaseBase):
|
||||
pass
|
||||
|
||||
class DocumentBase(BaseModel):
|
||||
file_name: str
|
||||
file_path: str
|
||||
file_hash: str
|
||||
file_size: int
|
||||
content_type: str
|
||||
|
||||
class DocumentCreate(DocumentBase):
|
||||
knowledge_base_id: int
|
||||
|
||||
class DocumentUploadBase(BaseModel):
|
||||
file_name: str
|
||||
file_hash: str
|
||||
file_size: int
|
||||
content_type: str
|
||||
temp_path: str
|
||||
status: str = "pending"
|
||||
error_message: Optional[str] = None
|
||||
|
||||
class DocumentUploadCreate(DocumentUploadBase):
|
||||
knowledge_base_id: int
|
||||
|
||||
class DocumentUploadResponse(DocumentUploadBase):
|
||||
id: int
|
||||
created_at: datetime
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
class ProcessingTaskBase(BaseModel):
|
||||
status: str
|
||||
error_message: Optional[str] = None
|
||||
|
||||
class ProcessingTaskCreate(ProcessingTaskBase):
|
||||
document_id: int
|
||||
knowledge_base_id: int
|
||||
|
||||
class ProcessingTask(ProcessingTaskBase):
|
||||
id: int
|
||||
document_id: int
|
||||
knowledge_base_id: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
class DocumentResponse(DocumentBase):
|
||||
id: int
|
||||
knowledge_base_id: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
processing_tasks: List[ProcessingTask] = []
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
class KnowledgeBaseResponse(KnowledgeBaseBase):
|
||||
id: int
|
||||
user_id: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
documents: List[DocumentResponse] = []
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
class PreviewRequest(BaseModel):
|
||||
document_ids: List[int]
|
||||
chunk_size: int = 1000
|
||||
chunk_overlap: int = 200
|
||||
59
rag-web-ui/backend/app/schemas/testing.py
Normal file
59
rag-web-ui/backend/app/schemas/testing.py
Normal file
@@ -0,0 +1,59 @@
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class TestItemEntry(BaseModel):
|
||||
id: str
|
||||
content: str
|
||||
|
||||
|
||||
class TestCaseEntry(BaseModel):
|
||||
id: str
|
||||
item_id: str
|
||||
operation_steps: List[str]
|
||||
test_content: str
|
||||
expected_result_placeholder: str
|
||||
|
||||
|
||||
class ExpectedResultEntry(BaseModel):
|
||||
id: str
|
||||
case_id: str
|
||||
result: str
|
||||
|
||||
|
||||
class StepLogEntry(BaseModel):
|
||||
step_name: str
|
||||
input_summary: str
|
||||
output_summary: str
|
||||
success: bool
|
||||
fallback_used: bool
|
||||
duration_ms: float
|
||||
|
||||
|
||||
class TestingPipelineRequest(BaseModel):
|
||||
requirement_text: str = Field(..., min_length=1)
|
||||
requirement_type: Optional[str] = None
|
||||
knowledge_base_ids: List[int] = []
|
||||
retrieval_top_k: int = Field(default=8, ge=1, le=20)
|
||||
knowledge_context: Optional[str] = None
|
||||
use_model_generation: bool = True
|
||||
max_items_per_group: int = Field(default=12, ge=4, le=30)
|
||||
cases_per_item: int = Field(default=2, ge=1, le=5)
|
||||
max_focus_points: int = Field(default=6, ge=3, le=12)
|
||||
max_llm_calls: int = Field(default=10, ge=0, le=100)
|
||||
debug: bool = False
|
||||
|
||||
|
||||
class TestingPipelineResponse(BaseModel):
|
||||
trace_id: str
|
||||
requirement_type: str
|
||||
reason: str
|
||||
candidates: List[str]
|
||||
test_items: Dict[str, List[TestItemEntry]]
|
||||
test_cases: Dict[str, List[TestCaseEntry]]
|
||||
expected_results: Dict[str, List[ExpectedResultEntry]]
|
||||
formatted_output: str
|
||||
pipeline_summary: str
|
||||
knowledge_used: bool = False
|
||||
step_logs: List[StepLogEntry] = []
|
||||
9
rag-web-ui/backend/app/schemas/token.py
Normal file
9
rag-web-ui/backend/app/schemas/token.py
Normal file
@@ -0,0 +1,9 @@
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional
|
||||
|
||||
class Token(BaseModel):
|
||||
access_token: str
|
||||
token_type: str
|
||||
|
||||
class TokenPayload(BaseModel):
|
||||
sub: Optional[int] = None
|
||||
52
rag-web-ui/backend/app/schemas/tooling.py
Normal file
52
rag-web-ui/backend/app/schemas/tooling.py
Normal file
@@ -0,0 +1,52 @@
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class ToolDefinitionResponse(BaseModel):
|
||||
name: str
|
||||
version: str
|
||||
description: str
|
||||
input_schema: Dict[str, Any]
|
||||
output_schema: Dict[str, Any]
|
||||
|
||||
|
||||
class SRSToolCreateJobResponse(BaseModel):
|
||||
job_id: int
|
||||
status: str
|
||||
|
||||
|
||||
class SRSToolJobStatusResponse(BaseModel):
|
||||
job_id: int
|
||||
tool_name: str
|
||||
status: str
|
||||
error_message: Optional[str] = None
|
||||
extraction_id: Optional[int] = None
|
||||
started_at: Optional[datetime] = None
|
||||
completed_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class SRSToolRequirementItem(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
description: str
|
||||
priority: str
|
||||
acceptanceCriteria: List[str]
|
||||
sourceField: str
|
||||
sectionNumber: Optional[str] = None
|
||||
sectionTitle: Optional[str] = None
|
||||
requirementType: Optional[str] = None
|
||||
sortOrder: int
|
||||
|
||||
|
||||
class SRSToolResultResponse(BaseModel):
|
||||
jobId: int
|
||||
documentName: str
|
||||
generatedAt: str
|
||||
statistics: Dict[str, Any]
|
||||
requirements: List[SRSToolRequirementItem]
|
||||
|
||||
|
||||
class SRSToolRequirementsSaveRequest(BaseModel):
|
||||
requirements: List[SRSToolRequirementItem]
|
||||
23
rag-web-ui/backend/app/schemas/user.py
Normal file
23
rag-web-ui/backend/app/schemas/user.py
Normal file
@@ -0,0 +1,23 @@
|
||||
from pydantic import BaseModel, EmailStr
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
|
||||
class UserBase(BaseModel):
|
||||
email: EmailStr
|
||||
username: str
|
||||
is_active: bool = True
|
||||
is_superuser: bool = False
|
||||
|
||||
class UserCreate(UserBase):
|
||||
password: str
|
||||
|
||||
class UserUpdate(UserBase):
|
||||
password: Optional[str] = None
|
||||
|
||||
class UserResponse(UserBase):
|
||||
id: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
0
rag-web-ui/backend/app/services/__init__.py
Normal file
0
rag-web-ui/backend/app/services/__init__.py
Normal file
61
rag-web-ui/backend/app/services/api_key.py
Normal file
61
rag-web-ui/backend/app/services/api_key.py
Normal file
@@ -0,0 +1,61 @@
|
||||
from typing import List, Optional
|
||||
from datetime import datetime
|
||||
import secrets
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models.api_key import APIKey
|
||||
from app.schemas.api_key import APIKeyCreate, APIKeyUpdate
|
||||
|
||||
class APIKeyService:
|
||||
@staticmethod
|
||||
def get_api_keys(db: Session, user_id: int, skip: int = 0, limit: int = 100) -> List[APIKey]:
|
||||
return (
|
||||
db.query(APIKey)
|
||||
.filter(APIKey.user_id == user_id)
|
||||
.offset(skip)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def create_api_key(db: Session, user_id: int, name: str) -> APIKey:
|
||||
api_key = APIKey(
|
||||
key=f"sk-{secrets.token_hex(32)}",
|
||||
name=name,
|
||||
user_id=user_id,
|
||||
is_active=True
|
||||
)
|
||||
db.add(api_key)
|
||||
db.commit()
|
||||
db.refresh(api_key)
|
||||
return api_key
|
||||
|
||||
@staticmethod
|
||||
def get_api_key(db: Session, api_key_id: int) -> Optional[APIKey]:
|
||||
return db.query(APIKey).filter(APIKey.id == api_key_id).first()
|
||||
|
||||
@staticmethod
|
||||
def get_api_key_by_key(db: Session, key: str) -> Optional[APIKey]:
|
||||
return db.query(APIKey).filter(APIKey.key == key).first()
|
||||
|
||||
@staticmethod
|
||||
def update_api_key(db: Session, api_key: APIKey, update_data: APIKeyUpdate) -> APIKey:
|
||||
for field, value in update_data.model_dump(exclude_unset=True).items():
|
||||
setattr(api_key, field, value)
|
||||
db.add(api_key)
|
||||
db.commit()
|
||||
db.refresh(api_key)
|
||||
return api_key
|
||||
|
||||
@staticmethod
|
||||
def delete_api_key(db: Session, api_key: APIKey) -> None:
|
||||
db.delete(api_key)
|
||||
db.commit()
|
||||
|
||||
@staticmethod
|
||||
def update_last_used(db: Session, api_key: APIKey) -> APIKey:
|
||||
api_key.last_used_at = datetime.utcnow()
|
||||
db.add(api_key)
|
||||
db.commit()
|
||||
db.refresh(api_key)
|
||||
return api_key
|
||||
532
rag-web-ui/backend/app/services/chat_service.py
Normal file
532
rag-web-ui/backend/app/services/chat_service.py
Normal file
@@ -0,0 +1,532 @@
|
||||
import base64
|
||||
import json
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from typing import Any, AsyncGenerator, Dict, List, Optional
|
||||
|
||||
from app.core.config import settings
|
||||
from app.models.chat import Message
|
||||
from app.models.knowledge import Document, KnowledgeBase
|
||||
from app.services.embedding.embedding_factory import EmbeddingsFactory
|
||||
from app.services.fusion_prompts import (
|
||||
GENERAL_CHAT_PROMPT_TEMPLATE,
|
||||
GRAPH_GLOBAL_PROMPT_TEMPLATE,
|
||||
GRAPH_LOCAL_PROMPT_TEMPLATE,
|
||||
HYBRID_RAG_PROMPT_TEMPLATE,
|
||||
)
|
||||
from app.services.graph.graphrag_adapter import GraphRAGAdapter
|
||||
from app.services.intent_router import route_intent
|
||||
from app.services.llm.llm_factory import LLMFactory
|
||||
from app.services.reranker.external_api import ExternalRerankerClient
|
||||
from app.services.retrieval.multi_kb_retriever import MultiKBRetriever, format_retrieval_context
|
||||
from app.services.testing_pipeline.pipeline import run_testing_pipeline
|
||||
from app.services.testing_pipeline.rules import REQUIREMENT_TYPES
|
||||
from app.services.vector_store import VectorStoreFactory
|
||||
|
||||
|
||||
TESTING_TARGET_KEYWORDS = [
|
||||
"测试项",
|
||||
"测试用例",
|
||||
"预期成果",
|
||||
"需求类型",
|
||||
"测试分解",
|
||||
"分解",
|
||||
"正常测试",
|
||||
"异常测试",
|
||||
"测试充分性",
|
||||
]
|
||||
|
||||
TESTING_ACTION_KEYWORDS = [
|
||||
"生成",
|
||||
"输出",
|
||||
"给出",
|
||||
"写",
|
||||
"编写",
|
||||
"设计",
|
||||
"整理",
|
||||
"列出",
|
||||
"提供",
|
||||
"制定",
|
||||
]
|
||||
|
||||
TYPE_ALIAS_MAP = {
|
||||
"接口测试": "外部接口测试",
|
||||
"ui测试": "人机交互界面测试",
|
||||
"界面测试": "人机交互界面测试",
|
||||
"恢复测试": "恢复性测试",
|
||||
"可靠性": "可靠性测试",
|
||||
"安全性": "安全性测试",
|
||||
"边界": "边界测试",
|
||||
"安装": "安装性测试",
|
||||
"互操作": "互操作性测试",
|
||||
"敏感性": "敏感性测试",
|
||||
"充分性": "测试充分性要求",
|
||||
}
|
||||
|
||||
|
||||
def _escape_stream_text(text: str) -> str:
|
||||
return text.replace('"', '\\"').replace("\n", "\\n")
|
||||
|
||||
|
||||
def _extract_stream_text(chunk: Any) -> str:
|
||||
content = getattr(chunk, "content", chunk)
|
||||
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
if isinstance(content, list):
|
||||
parts: List[str] = []
|
||||
for item in content:
|
||||
if isinstance(item, str):
|
||||
parts.append(item)
|
||||
elif isinstance(item, dict):
|
||||
maybe_text = item.get("text")
|
||||
if isinstance(maybe_text, str):
|
||||
parts.append(maybe_text)
|
||||
else:
|
||||
parts.append(str(item))
|
||||
return "".join(parts)
|
||||
|
||||
return str(content)
|
||||
|
||||
|
||||
def _preview_rows(rows: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
preview = []
|
||||
for row in rows[:10]:
|
||||
doc = row["document"]
|
||||
metadata = doc.metadata or {}
|
||||
preview.append(
|
||||
{
|
||||
"kb_id": row.get("kb_id"),
|
||||
"source": metadata.get("source") or metadata.get("file_name") or "unknown",
|
||||
"chunk_id": metadata.get("chunk_id") or "unknown",
|
||||
"score": row.get("final_score", 0),
|
||||
"reranker_score": row.get("reranker_score"),
|
||||
}
|
||||
)
|
||||
return preview
|
||||
|
||||
|
||||
def _context_rows(rows: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
context_rows: List[Dict[str, Any]] = []
|
||||
for row in rows:
|
||||
doc = row["document"]
|
||||
metadata = dict(doc.metadata or {})
|
||||
|
||||
if "kb_id" not in metadata and row.get("kb_id") is not None:
|
||||
metadata["kb_id"] = row.get("kb_id")
|
||||
metadata.setdefault("retrieval_score", row.get("final_score", 0))
|
||||
if row.get("reranker_score") is not None:
|
||||
metadata.setdefault("reranker_score", row.get("reranker_score"))
|
||||
|
||||
context_rows.append(
|
||||
{
|
||||
"page_content": doc.page_content.strip(),
|
||||
"metadata": metadata,
|
||||
}
|
||||
)
|
||||
return context_rows
|
||||
|
||||
|
||||
def _build_local_graph_context_fallback(rows: List[Dict[str, Any]]) -> str:
|
||||
entities = set()
|
||||
relations: List[Dict[str, Any]] = []
|
||||
evidences: List[str] = []
|
||||
|
||||
for row in rows:
|
||||
doc = row["document"]
|
||||
metadata = doc.metadata or {}
|
||||
|
||||
for ent in metadata.get("extracted_entities", []):
|
||||
entities.add(str(ent))
|
||||
|
||||
for rel in metadata.get("extracted_relations", []):
|
||||
if isinstance(rel, dict):
|
||||
relations.append(rel)
|
||||
|
||||
evidences.append(doc.page_content.strip())
|
||||
|
||||
entity_block = "\n".join(f"- {name}" for name in sorted(entities)[:80]) or "- 暂无结构化实体，已使用向量检索回退。"
|
||||
|
||||
relation_lines: List[str] = []
|
||||
for rel in relations[:120]:
|
||||
src = rel.get("source") or rel.get("src") or rel.get("src_id") or "UNKNOWN"
|
||||
tgt = rel.get("target") or rel.get("tgt") or rel.get("tgt_id") or "UNKNOWN"
|
||||
rel_type = rel.get("type") or rel.get("relation_type") or "其他"
|
||||
desc = rel.get("description") or ""
|
||||
relation_lines.append(f"- {src} -> {tgt} | 类型={rel_type} | 说明={desc}")
|
||||
|
||||
relation_block = "\n".join(relation_lines) or "- 暂无结构化关系，已使用证据片段回答。"
|
||||
|
||||
evidence_block = "\n\n".join(
|
||||
f"[证据{i}] {snippet}" for i, snippet in enumerate(evidences[:8], start=1)
|
||||
)
|
||||
if not evidence_block:
|
||||
evidence_block = "无可用证据。"
|
||||
|
||||
return (
|
||||
"实体列表:\n"
|
||||
f"{entity_block}\n\n"
|
||||
"关系列表:\n"
|
||||
f"{relation_block}\n\n"
|
||||
"原文证据:\n"
|
||||
f"{evidence_block}"
|
||||
)
|
||||
|
||||
|
||||
def _build_global_community_context_fallback(rows: List[Dict[str, Any]]) -> str:
|
||||
groups: Dict[str, List[str]] = defaultdict(list)
|
||||
|
||||
for row in rows:
|
||||
doc = row["document"]
|
||||
metadata = doc.metadata or {}
|
||||
community_ids = metadata.get("community_ids") or []
|
||||
|
||||
if isinstance(community_ids, list) and community_ids:
|
||||
keys = [str(item) for item in community_ids]
|
||||
else:
|
||||
source = metadata.get("source") or metadata.get("file_name") or "unknown"
|
||||
keys = [f"source:{source}"]
|
||||
|
||||
for key in keys:
|
||||
groups[key].append(doc.page_content.strip())
|
||||
|
||||
if not groups:
|
||||
return "暂无社区摘要数据，已回退为基于证据片段的全局总结。"
|
||||
|
||||
lines: List[str] = []
|
||||
for idx, (community_id, snippets) in enumerate(groups.items(), start=1):
|
||||
merged = " ".join(snippets[:3])
|
||||
lines.append(f"社区{idx} ({community_id}) 摘要: {merged}")
|
||||
|
||||
return "\n\n".join(lines)
|
||||
|
||||
|
||||
async def _build_kb_vector_stores(db: Any, knowledge_bases: List[KnowledgeBase]) -> List[Dict[str, Any]]:
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
kb_vector_stores: List[Dict[str, Any]] = []
|
||||
|
||||
for kb in knowledge_bases:
|
||||
documents = db.query(Document).filter(Document.knowledge_base_id == kb.id).all()
|
||||
if not documents:
|
||||
continue
|
||||
|
||||
store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{kb.id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
kb_vector_stores.append({"kb_id": kb.id, "store": store})
|
||||
|
||||
return kb_vector_stores
|
||||
|
||||
|
||||
def _build_reranker_client() -> ExternalRerankerClient:
|
||||
return ExternalRerankerClient(
|
||||
api_url=settings.RERANKER_API_URL,
|
||||
api_key=settings.RERANKER_API_KEY,
|
||||
model=settings.RERANKER_MODEL,
|
||||
timeout_seconds=settings.RERANKER_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
|
||||
def _is_testing_generation_request(query: str) -> bool:
|
||||
text = (query or "").strip()
|
||||
if not text:
|
||||
return False
|
||||
|
||||
normalized = text.lower()
|
||||
if normalized.startswith("/testing"):
|
||||
return True
|
||||
|
||||
if any(
|
||||
token in normalized
|
||||
for token in (
|
||||
"testing_orchestrator",
|
||||
"testing-orchestrator",
|
||||
"identify_requirement_type",
|
||||
"identify-requirement-type",
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
has_target = any(keyword in text for keyword in TESTING_TARGET_KEYWORDS)
|
||||
has_action = any(keyword in text for keyword in TESTING_ACTION_KEYWORDS)
|
||||
if has_target and has_action:
|
||||
return True
|
||||
|
||||
if any(keyword in text for keyword in ("测试项", "测试用例", "预期成果")):
|
||||
if re.search(r"(请|帮|给|麻烦).{0,12}(写|生成|设计|整理|编写|列出|提供|制定)", text):
|
||||
return True
|
||||
if text.startswith(("生成", "编写", "设计", "整理", "输出", "列出", "提供", "制定")):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _extract_requirement_type_from_query(query: str) -> Optional[str]:
|
||||
text = (query or "").strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
for req_type in REQUIREMENT_TYPES:
|
||||
if req_type in text:
|
||||
return req_type
|
||||
|
||||
lowered = text.lower()
|
||||
for alias, req_type in TYPE_ALIAS_MAP.items():
|
||||
if alias in text or alias in lowered:
|
||||
return req_type
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def generate_response(
|
||||
query: str,
|
||||
messages: dict,
|
||||
knowledge_base_ids: List[int],
|
||||
chat_id: int,
|
||||
db: Any,
|
||||
) -> AsyncGenerator[str, None]:
|
||||
try:
|
||||
user_message = Message(content=query, role="user", chat_id=chat_id)
|
||||
db.add(user_message)
|
||||
db.commit()
|
||||
|
||||
bot_message = Message(content="", role="assistant", chat_id=chat_id)
|
||||
db.add(bot_message)
|
||||
db.commit()
|
||||
|
||||
if _is_testing_generation_request(query):
|
||||
explicit_type = _extract_requirement_type_from_query(query)
|
||||
|
||||
retrieval_rows: List[Dict[str, Any]] = []
|
||||
knowledge_context = ""
|
||||
kb_vector_stores = []
|
||||
if knowledge_base_ids:
|
||||
testing_kbs = (
|
||||
db.query(KnowledgeBase)
|
||||
.filter(KnowledgeBase.id.in_(knowledge_base_ids))
|
||||
.all()
|
||||
)
|
||||
kb_vector_stores = await _build_kb_vector_stores(db, testing_kbs)
|
||||
|
||||
if kb_vector_stores:
|
||||
testing_retriever = MultiKBRetriever(
|
||||
reranker_weight=settings.RERANKER_WEIGHT,
|
||||
)
|
||||
retrieval_rows = await testing_retriever.retrieve(
|
||||
query=query,
|
||||
kb_vector_stores=kb_vector_stores,
|
||||
fetch_k_per_kb=16,
|
||||
top_k=8,
|
||||
)
|
||||
if retrieval_rows:
|
||||
knowledge_context = format_retrieval_context(retrieval_rows)
|
||||
|
||||
pipeline_result = run_testing_pipeline(
|
||||
user_requirement_text=query,
|
||||
requirement_type_input=explicit_type,
|
||||
debug=True,
|
||||
knowledge_context=knowledge_context,
|
||||
use_model_generation=True,
|
||||
max_items_per_group=6,
|
||||
cases_per_item=1,
|
||||
max_focus_points=6,
|
||||
max_llm_calls=2,
|
||||
)
|
||||
|
||||
context_payload = {
|
||||
"route": {
|
||||
"intent": "TESTING",
|
||||
"reason": "命中测试生成意图，已自动调用测试工具链。",
|
||||
},
|
||||
"intent": "TESTING",
|
||||
"skill_profile": "testing-orchestrator",
|
||||
"tool_chain": [
|
||||
"identify-requirement-type",
|
||||
"decompose-test-items",
|
||||
"generate-test-cases",
|
||||
"build_expected_results",
|
||||
"format_output",
|
||||
],
|
||||
"selected_chain": "TESTING_PIPELINE",
|
||||
"graph_used": False,
|
||||
"reranker_enabled": False,
|
||||
"retrieval_preview": _preview_rows(retrieval_rows),
|
||||
"context": _context_rows(retrieval_rows),
|
||||
"testing_pipeline": {
|
||||
"trace_id": pipeline_result.get("trace_id"),
|
||||
"requirement_type": pipeline_result.get("requirement_type"),
|
||||
"candidates": pipeline_result.get("candidates", []),
|
||||
"pipeline_summary": pipeline_result.get("pipeline_summary", ""),
|
||||
"knowledge_used": pipeline_result.get("knowledge_used", False),
|
||||
"step_logs": pipeline_result.get("step_logs", []),
|
||||
},
|
||||
}
|
||||
|
||||
escaped_context = json.dumps(context_payload, ensure_ascii=False)
|
||||
base64_context = base64.b64encode(escaped_context.encode()).decode()
|
||||
separator = "__LLM_RESPONSE__"
|
||||
|
||||
full_response = f"{base64_context}{separator}"
|
||||
yield f'0:"{base64_context}{separator}"\n'
|
||||
|
||||
rendered_text = pipeline_result.get("formatted_output", "").strip()
|
||||
if not rendered_text:
|
||||
rendered_text = "未生成测试内容，请补充更明确的需求后重试。"
|
||||
|
||||
full_response += rendered_text
|
||||
yield f'0:"{_escape_stream_text(rendered_text)}"\n'
|
||||
yield 'd:{"finishReason":"stop","usage":{"promptTokens":0,"completionTokens":0}}\n'
|
||||
|
||||
bot_message.content = full_response
|
||||
db.commit()
|
||||
return
|
||||
|
||||
knowledge_bases = (
|
||||
db.query(KnowledgeBase)
|
||||
.filter(KnowledgeBase.id.in_(knowledge_base_ids))
|
||||
.all()
|
||||
)
|
||||
kb_ids = [kb.id for kb in knowledge_bases]
|
||||
|
||||
llm = LLMFactory.create()
|
||||
decision = await route_intent(llm=llm, query=query, messages=messages)
|
||||
intent = decision["intent"]
|
||||
|
||||
kb_vector_stores = await _build_kb_vector_stores(db, knowledge_bases)
|
||||
if intent in {"B", "C", "D"} and not kb_vector_stores:
|
||||
intent = "A"
|
||||
decision = {
|
||||
"intent": "A",
|
||||
"reason": "未发现可用知识库向量集合，已降级为通用对话路。",
|
||||
}
|
||||
|
||||
reranker_client = _build_reranker_client()
|
||||
retriever = MultiKBRetriever(
|
||||
reranker_client=reranker_client,
|
||||
reranker_weight=settings.RERANKER_WEIGHT,
|
||||
)
|
||||
|
||||
retrieval_rows: List[Dict[str, Any]] = []
|
||||
graph_used = False
|
||||
selected_chain = intent
|
||||
prompt_text = ""
|
||||
|
||||
if intent == "A":
|
||||
prompt_text = GENERAL_CHAT_PROMPT_TEMPLATE.format(query=query)
|
||||
|
||||
elif intent == "B":
|
||||
retrieval_rows = await retriever.retrieve(
|
||||
query=query,
|
||||
kb_vector_stores=kb_vector_stores,
|
||||
fetch_k_per_kb=16,
|
||||
top_k=12,
|
||||
)
|
||||
context = format_retrieval_context(retrieval_rows) or "无可用证据。"
|
||||
prompt_text = HYBRID_RAG_PROMPT_TEMPLATE.format(query=query, context=context)
|
||||
|
||||
elif intent == "C":
|
||||
graph_context = ""
|
||||
used_kb_ids: List[int] = []
|
||||
if settings.GRAPHRAG_ENABLED and kb_ids:
|
||||
try:
|
||||
adapter = GraphRAGAdapter()
|
||||
graph_context, used_kb_ids = await adapter.local_context_multi(
|
||||
kb_ids,
|
||||
query,
|
||||
top_k=settings.GRAPHRAG_LOCAL_TOP_K,
|
||||
level=settings.GRAPHRAG_QUERY_LEVEL,
|
||||
)
|
||||
graph_used = bool(graph_context)
|
||||
except Exception:
|
||||
graph_context = ""
|
||||
|
||||
if not graph_context:
|
||||
retrieval_rows = await retriever.retrieve(
|
||||
query=query,
|
||||
kb_vector_stores=kb_vector_stores,
|
||||
fetch_k_per_kb=18,
|
||||
top_k=14,
|
||||
)
|
||||
graph_context = _build_local_graph_context_fallback(retrieval_rows)
|
||||
selected_chain = "C_fallback_B"
|
||||
|
||||
else:
|
||||
selected_chain = "C_graph"
|
||||
|
||||
prompt_text = GRAPH_LOCAL_PROMPT_TEMPLATE.format(
|
||||
query=query,
|
||||
graph_context=graph_context,
|
||||
)
|
||||
|
||||
else:
|
||||
community_context = ""
|
||||
if settings.GRAPHRAG_ENABLED and kb_ids:
|
||||
try:
|
||||
adapter = GraphRAGAdapter()
|
||||
community_context, used_kb_ids = await adapter.global_context_multi(
|
||||
kb_ids,
|
||||
query,
|
||||
level=settings.GRAPHRAG_QUERY_LEVEL,
|
||||
)
|
||||
graph_used = bool(community_context)
|
||||
except Exception:
|
||||
community_context = ""
|
||||
|
||||
if not community_context:
|
||||
retrieval_rows = await retriever.retrieve(
|
||||
query=query,
|
||||
kb_vector_stores=kb_vector_stores,
|
||||
fetch_k_per_kb=20,
|
||||
top_k=14,
|
||||
)
|
||||
community_context = _build_global_community_context_fallback(retrieval_rows)
|
||||
selected_chain = "D_fallback_B"
|
||||
else:
|
||||
selected_chain = "D_graph"
|
||||
|
||||
prompt_text = GRAPH_GLOBAL_PROMPT_TEMPLATE.format(
|
||||
query=query,
|
||||
community_context=community_context,
|
||||
)
|
||||
|
||||
context_payload = {
|
||||
"route": decision,
|
||||
"intent": intent,
|
||||
"selected_chain": selected_chain,
|
||||
"graph_used": graph_used,
|
||||
"reranker_enabled": reranker_client.enabled,
|
||||
"retrieval_preview": _preview_rows(retrieval_rows),
|
||||
"context": _context_rows(retrieval_rows),
|
||||
}
|
||||
escaped_context = json.dumps(context_payload, ensure_ascii=False)
|
||||
base64_context = base64.b64encode(escaped_context.encode()).decode()
|
||||
separator = "__LLM_RESPONSE__"
|
||||
|
||||
full_response = f"{base64_context}{separator}"
|
||||
yield f'0:"{base64_context}{separator}"\n'
|
||||
|
||||
async for chunk in llm.astream(prompt_text):
|
||||
text = _extract_stream_text(chunk)
|
||||
if not text:
|
||||
continue
|
||||
full_response += text
|
||||
yield f'0:"{_escape_stream_text(text)}"\n'
|
||||
|
||||
yield 'd:{"finishReason":"stop","usage":{"promptTokens":0,"completionTokens":0}}\n'
|
||||
|
||||
bot_message.content = full_response
|
||||
db.commit()
|
||||
|
||||
except Exception as e:
|
||||
error_message = f"Error generating response: {str(e)}"
|
||||
print(error_message)
|
||||
yield "3:{text}\n".format(text=error_message)
|
||||
|
||||
if "bot_message" in locals():
|
||||
bot_message.content = error_message
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
69
rag-web-ui/backend/app/services/chunk_record.py
Normal file
69
rag-web-ui/backend/app/services/chunk_record.py
Normal file
@@ -0,0 +1,69 @@
|
||||
from typing import Optional, List, Dict, Set
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.orm import Session
|
||||
from app.core.config import settings
|
||||
from app.models.knowledge import DocumentChunk
|
||||
import json
|
||||
|
||||
class ChunkRecord:
|
||||
"""Manages chunk-level record keeping for incremental updates"""
|
||||
def __init__(self, kb_id: int):
|
||||
self.kb_id = kb_id
|
||||
self.engine = create_engine(settings.get_database_url)
|
||||
|
||||
def list_chunks(self, file_name: Optional[str] = None) -> Set[str]:
|
||||
"""List all chunk hashes for the given file"""
|
||||
with Session(self.engine) as session:
|
||||
query = session.query(DocumentChunk.hash).filter(
|
||||
DocumentChunk.kb_id == self.kb_id
|
||||
)
|
||||
|
||||
if file_name:
|
||||
query = query.filter(DocumentChunk.file_name == file_name)
|
||||
|
||||
return {row[0] for row in query.all()}
|
||||
|
||||
def add_chunks(self, chunks: List[Dict]):
|
||||
"""Add new chunks to the database"""
|
||||
if not chunks:
|
||||
return
|
||||
|
||||
with Session(self.engine) as session:
|
||||
for chunk_data in chunks:
|
||||
chunk = DocumentChunk(
|
||||
id=chunk_data['id'],
|
||||
kb_id=chunk_data['kb_id'],
|
||||
document_id=chunk_data['document_id'],
|
||||
file_name=chunk_data['file_name'],
|
||||
chunk_metadata=chunk_data['metadata'],
|
||||
hash=chunk_data['hash']
|
||||
)
|
||||
session.merge(chunk) # Use merge instead of add to handle updates
|
||||
session.commit()
|
||||
|
||||
def delete_chunks(self, chunk_ids: List[str]):
|
||||
"""Delete chunks by their IDs"""
|
||||
if not chunk_ids:
|
||||
return
|
||||
|
||||
with Session(self.engine) as session:
|
||||
session.query(DocumentChunk).filter(
|
||||
DocumentChunk.kb_id == self.kb_id,
|
||||
DocumentChunk.id.in_(chunk_ids)
|
||||
).delete(synchronize_session=False)
|
||||
session.commit()
|
||||
|
||||
def get_deleted_chunks(self, current_hashes: Set[str], file_name: Optional[str] = None) -> List[str]:
|
||||
"""Get IDs of chunks that no longer exist in the current version"""
|
||||
with Session(self.engine) as session:
|
||||
query = session.query(DocumentChunk.id).filter(
|
||||
DocumentChunk.kb_id == self.kb_id
|
||||
)
|
||||
|
||||
if file_name:
|
||||
query = query.filter(DocumentChunk.file_name == file_name)
|
||||
|
||||
if current_hashes:
|
||||
query = query.filter(DocumentChunk.hash.notin_(current_hashes))
|
||||
|
||||
return [row[0] for row in query.all()]
|
||||
582
rag-web-ui/backend/app/services/document_processor.py
Normal file
582
rag-web-ui/backend/app/services/document_processor.py
Normal file
@@ -0,0 +1,582 @@
|
||||
import logging
|
||||
import os
|
||||
import hashlib
|
||||
import tempfile
|
||||
import traceback
|
||||
import json
|
||||
from app.db.session import SessionLocal
|
||||
from io import BytesIO
|
||||
from typing import Optional, List, Dict, Any
|
||||
from fastapi import UploadFile
|
||||
from langchain_community.document_loaders import (
|
||||
PyPDFLoader,
|
||||
Docx2txtLoader,
|
||||
UnstructuredMarkdownLoader,
|
||||
TextLoader
|
||||
)
|
||||
from langchain.text_splitter import RecursiveCharacterTextSplitter
|
||||
from langchain_core.documents import Document as LangchainDocument
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm import Session
|
||||
from app.core.config import settings
|
||||
from app.core.minio import get_minio_client
|
||||
from app.models.knowledge import ProcessingTask, Document, DocumentChunk
|
||||
from app.services.chunk_record import ChunkRecord
|
||||
from minio.error import MinioException
|
||||
from minio.commonconfig import CopySource
|
||||
from app.services.vector_store import VectorStoreFactory
|
||||
from app.services.embedding.embedding_factory import EmbeddingsFactory
|
||||
|
||||
class UploadResult(BaseModel):
|
||||
file_path: str
|
||||
file_name: str
|
||||
file_size: int
|
||||
content_type: str
|
||||
file_hash: str
|
||||
|
||||
class TextChunk(BaseModel):
|
||||
content: str
|
||||
metadata: Optional[Dict] = None
|
||||
|
||||
class PreviewResult(BaseModel):
|
||||
chunks: List[TextChunk]
|
||||
total_chunks: int
|
||||
|
||||
|
||||
def _estimate_token_count(text: str) -> int:
|
||||
# Lightweight estimation without adding tokenizer dependencies.
|
||||
return len(text)
|
||||
|
||||
|
||||
def _build_enriched_chunk_metadata(
|
||||
*,
|
||||
source_metadata: Optional[Dict[str, Any]],
|
||||
chunk_id: str,
|
||||
file_name: str,
|
||||
file_path: str,
|
||||
kb_id: int,
|
||||
document_id: int,
|
||||
chunk_index: int,
|
||||
chunk_text: str,
|
||||
) -> Dict[str, Any]:
|
||||
source_metadata = source_metadata or {}
|
||||
token_count = _estimate_token_count(chunk_text)
|
||||
|
||||
return {
|
||||
**source_metadata,
|
||||
"source": file_name,
|
||||
"chunk_id": chunk_id,
|
||||
"file_name": file_name,
|
||||
"file_path": file_path,
|
||||
"kb_id": kb_id,
|
||||
"document_id": document_id,
|
||||
"chunk_index": chunk_index,
|
||||
"chunk_text": chunk_text,
|
||||
"token_count": token_count,
|
||||
"language": source_metadata.get("language", "zh"),
|
||||
"source_type": "document",
|
||||
"mission_phase": source_metadata.get("mission_phase"),
|
||||
"section_title": source_metadata.get("section_title"),
|
||||
"publish_time": source_metadata.get("publish_time"),
|
||||
# Keep graph-linked fields for future graph/vector federation.
|
||||
"extracted_entities": source_metadata.get("extracted_entities", []),
|
||||
"extracted_entity_types": source_metadata.get("extracted_entity_types", []),
|
||||
"extracted_relations": source_metadata.get("extracted_relations", []),
|
||||
"graph_node_ids": source_metadata.get("graph_node_ids", []),
|
||||
"graph_edge_ids": source_metadata.get("graph_edge_ids", []),
|
||||
"community_ids": source_metadata.get("community_ids", []),
|
||||
}
|
||||
|
||||
|
||||
def _sanitize_metadata_for_vector_store(metadata: Optional[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Normalize metadata to satisfy Chroma's strict metadata constraints."""
|
||||
if not metadata:
|
||||
return {}
|
||||
|
||||
sanitized: Dict[str, Any] = {}
|
||||
scalar_types = (str, int, float, bool)
|
||||
|
||||
for key, value in metadata.items():
|
||||
if value is None:
|
||||
continue
|
||||
|
||||
if isinstance(value, scalar_types):
|
||||
sanitized[key] = value
|
||||
continue
|
||||
|
||||
if isinstance(value, list):
|
||||
primitive_items = [item for item in value if isinstance(item, scalar_types)]
|
||||
if primitive_items:
|
||||
sanitized[key] = primitive_items
|
||||
elif value:
|
||||
sanitized[key] = json.dumps(value, ensure_ascii=False)
|
||||
continue
|
||||
|
||||
if isinstance(value, dict):
|
||||
sanitized[key] = json.dumps(value, ensure_ascii=False)
|
||||
continue
|
||||
|
||||
sanitized[key] = str(value)
|
||||
|
||||
return sanitized
|
||||
|
||||
async def process_document(file_path: str, file_name: str, kb_id: int, document_id: int, chunk_size: int = 1000, chunk_overlap: int = 200) -> None:
|
||||
"""Process document and store in vector database with incremental updates"""
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
preview_result = await preview_document(file_path, chunk_size, chunk_overlap)
|
||||
|
||||
# Initialize embeddings
|
||||
logger.info("Initializing OpenAI embeddings...")
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
|
||||
logger.info(f"Initializing vector store with collection: kb_{kb_id}")
|
||||
vector_store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{kb_id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
|
||||
# Initialize chunk record manager
|
||||
chunk_manager = ChunkRecord(kb_id)
|
||||
|
||||
# Get existing chunk hashes for this file
|
||||
existing_hashes = chunk_manager.list_chunks(file_name)
|
||||
|
||||
# Prepare new chunks
|
||||
new_chunks = []
|
||||
current_hashes = set()
|
||||
documents_to_update = []
|
||||
|
||||
for i, chunk in enumerate(preview_result.chunks):
|
||||
# Calculate chunk hash
|
||||
chunk_hash = hashlib.sha256(
|
||||
(chunk.content + str(chunk.metadata)).encode()
|
||||
).hexdigest()
|
||||
current_hashes.add(chunk_hash)
|
||||
|
||||
# Skip if chunk hasn't changed
|
||||
if chunk_hash in existing_hashes:
|
||||
continue
|
||||
|
||||
# Create unique ID for the chunk
|
||||
chunk_id = hashlib.sha256(
|
||||
f"{kb_id}:{file_name}:{chunk_hash}".encode()
|
||||
).hexdigest()
|
||||
|
||||
metadata = _build_enriched_chunk_metadata(
|
||||
source_metadata=chunk.metadata,
|
||||
chunk_id=chunk_id,
|
||||
file_name=file_name,
|
||||
file_path=file_path,
|
||||
kb_id=kb_id,
|
||||
document_id=document_id,
|
||||
chunk_index=i,
|
||||
chunk_text=chunk.content,
|
||||
)
|
||||
vector_metadata = _sanitize_metadata_for_vector_store(metadata)
|
||||
|
||||
new_chunks.append({
|
||||
"id": chunk_id,
|
||||
"kb_id": kb_id,
|
||||
"document_id": document_id,
|
||||
"file_name": file_name,
|
||||
"metadata": metadata,
|
||||
"hash": chunk_hash
|
||||
})
|
||||
|
||||
# Prepare document for vector store
|
||||
doc = LangchainDocument(
|
||||
page_content=chunk.content,
|
||||
metadata=vector_metadata
|
||||
)
|
||||
documents_to_update.append(doc)
|
||||
|
||||
# Add new chunks to database and vector store
|
||||
if new_chunks:
|
||||
logger.info(f"Adding {len(new_chunks)} new/updated chunks")
|
||||
chunk_manager.add_chunks(new_chunks)
|
||||
vector_store.add_documents(documents_to_update)
|
||||
if settings.GRAPHRAG_ENABLED:
|
||||
try:
|
||||
from app.services.graph.graphrag_adapter import GraphRAGAdapter
|
||||
|
||||
graph_adapter = GraphRAGAdapter()
|
||||
source_texts = [doc.page_content for doc in documents_to_update if doc.page_content.strip()]
|
||||
await graph_adapter.ingest_texts(kb_id, source_texts)
|
||||
logger.info("GraphRAG ingestion completed in incremental processing")
|
||||
except Exception as graph_exc:
|
||||
logger.error(f"GraphRAG ingestion failed in incremental processing: {graph_exc}")
|
||||
|
||||
# Delete removed chunks
|
||||
chunks_to_delete = chunk_manager.get_deleted_chunks(current_hashes, file_name)
|
||||
if chunks_to_delete:
|
||||
logger.info(f"Removing {len(chunks_to_delete)} deleted chunks")
|
||||
chunk_manager.delete_chunks(chunks_to_delete)
|
||||
vector_store.delete(chunks_to_delete)
|
||||
|
||||
logger.info("Document processing completed successfully")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing document: {str(e)}")
|
||||
raise
|
||||
|
||||
async def upload_document(file: UploadFile, kb_id: int) -> UploadResult:
|
||||
"""Step 1: Upload document to MinIO"""
|
||||
content = await file.read()
|
||||
file_size = len(content)
|
||||
|
||||
file_hash = hashlib.sha256(content).hexdigest()
|
||||
|
||||
# Clean and normalize filename
|
||||
file_name = "".join(c for c in file.filename if c.isalnum() or c in ('-', '_', '.')).strip()
|
||||
object_path = f"kb_{kb_id}/{file_name}"
|
||||
|
||||
content_types = {
|
||||
".pdf": "application/pdf",
|
||||
".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
".md": "text/markdown",
|
||||
".txt": "text/plain"
|
||||
}
|
||||
|
||||
_, ext = os.path.splitext(file_name)
|
||||
content_type = content_types.get(ext.lower(), "application/octet-stream")
|
||||
|
||||
# Upload to MinIO
|
||||
minio_client = get_minio_client()
|
||||
try:
|
||||
minio_client.put_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=object_path,
|
||||
data=BytesIO(content),
|
||||
length=file_size,
|
||||
content_type=content_type
|
||||
)
|
||||
except Exception as e:
|
||||
logging.error(f"Failed to upload file to MinIO: {str(e)}")
|
||||
raise
|
||||
|
||||
return UploadResult(
|
||||
file_path=object_path,
|
||||
file_name=file_name,
|
||||
file_size=file_size,
|
||||
content_type=content_type,
|
||||
file_hash=file_hash
|
||||
)
|
||||
|
||||
async def preview_document(file_path: str, chunk_size: int = 1000, chunk_overlap: int = 200) -> PreviewResult:
|
||||
"""Step 2: Generate preview chunks"""
|
||||
# Get file from MinIO
|
||||
minio_client = get_minio_client()
|
||||
_, ext = os.path.splitext(file_path)
|
||||
ext = ext.lower()
|
||||
|
||||
# Download to temp file
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=ext) as temp_file:
|
||||
minio_client.fget_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=file_path,
|
||||
file_path=temp_file.name
|
||||
)
|
||||
temp_path = temp_file.name
|
||||
|
||||
try:
|
||||
# Select appropriate loader
|
||||
if ext == ".pdf":
|
||||
loader = PyPDFLoader(temp_path)
|
||||
elif ext == ".docx":
|
||||
loader = Docx2txtLoader(temp_path)
|
||||
elif ext == ".md":
|
||||
loader = UnstructuredMarkdownLoader(temp_path)
|
||||
else: # Default to text loader
|
||||
loader = TextLoader(temp_path)
|
||||
|
||||
# Load and split the document
|
||||
documents = loader.load()
|
||||
text_splitter = RecursiveCharacterTextSplitter(
|
||||
chunk_size=chunk_size,
|
||||
chunk_overlap=chunk_overlap
|
||||
)
|
||||
chunks = text_splitter.split_documents(documents)
|
||||
|
||||
# Convert to preview format
|
||||
preview_chunks = [
|
||||
TextChunk(
|
||||
content=chunk.page_content,
|
||||
metadata=chunk.metadata
|
||||
)
|
||||
for chunk in chunks
|
||||
]
|
||||
|
||||
return PreviewResult(
|
||||
chunks=preview_chunks,
|
||||
total_chunks=len(chunks)
|
||||
)
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
async def process_document_background(
|
||||
temp_path: str,
|
||||
file_name: str,
|
||||
kb_id: int,
|
||||
task_id: int,
|
||||
db: Session = None,
|
||||
chunk_size: int = 1000,
|
||||
chunk_overlap: int = 200
|
||||
) -> None:
|
||||
"""Process document in background"""
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.info(f"Starting background processing for task {task_id}, file: {file_name}")
|
||||
|
||||
# if we don't pass in db, create a new database session
|
||||
if db is None:
|
||||
db = SessionLocal()
|
||||
should_close_db = True
|
||||
else:
|
||||
should_close_db = False
|
||||
|
||||
task = db.query(ProcessingTask).get(task_id)
|
||||
if not task:
|
||||
logger.error(f"Task {task_id} not found")
|
||||
return
|
||||
|
||||
minio_client = None
|
||||
local_temp_path = None
|
||||
|
||||
try:
|
||||
logger.info(f"Task {task_id}: Setting status to processing")
|
||||
task.status = "processing"
|
||||
db.commit()
|
||||
|
||||
# 1. 从临时目录下载文件
|
||||
minio_client = get_minio_client()
|
||||
try:
|
||||
local_temp_path = f"/tmp/temp_{task_id}_{file_name}" # 使用系统临时目录
|
||||
logger.info(f"Task {task_id}: Downloading file from MinIO: {temp_path} to {local_temp_path}")
|
||||
minio_client.fget_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=temp_path,
|
||||
file_path=local_temp_path
|
||||
)
|
||||
logger.info(f"Task {task_id}: File downloaded successfully")
|
||||
except MinioException as e:
|
||||
# Idempotent fallback: temp object may already be consumed by another task.
|
||||
# If the final document is already created, treat current task as completed.
|
||||
if "NoSuchKey" in str(e) and task.document_upload:
|
||||
existing_document = db.query(Document).filter(
|
||||
Document.knowledge_base_id == kb_id,
|
||||
Document.file_name == file_name,
|
||||
Document.file_hash == task.document_upload.file_hash,
|
||||
).first()
|
||||
if existing_document:
|
||||
logger.warning(
|
||||
f"Task {task_id}: Temp object missing but document already exists, "
|
||||
f"marking task as completed (document_id={existing_document.id})"
|
||||
)
|
||||
task.status = "completed"
|
||||
task.document_id = existing_document.id
|
||||
task.error_message = None
|
||||
task.document_upload.status = "completed"
|
||||
task.document_upload.error_message = None
|
||||
db.commit()
|
||||
return
|
||||
|
||||
error_msg = f"Failed to download temp file: {str(e)}"
|
||||
logger.error(f"Task {task_id}: {error_msg}")
|
||||
raise Exception(error_msg)
|
||||
|
||||
try:
|
||||
# 2. 加载和分块文档
|
||||
_, ext = os.path.splitext(file_name)
|
||||
ext = ext.lower()
|
||||
|
||||
logger.info(f"Task {task_id}: Loading document with extension {ext}")
|
||||
# 选择合适的加载器
|
||||
if ext == ".pdf":
|
||||
loader = PyPDFLoader(local_temp_path)
|
||||
elif ext == ".docx":
|
||||
loader = Docx2txtLoader(local_temp_path)
|
||||
elif ext == ".md":
|
||||
loader = UnstructuredMarkdownLoader(local_temp_path)
|
||||
else: # 默认使用文本加载器
|
||||
loader = TextLoader(local_temp_path)
|
||||
|
||||
logger.info(f"Task {task_id}: Loading document content")
|
||||
documents = loader.load()
|
||||
logger.info(f"Task {task_id}: Document loaded successfully")
|
||||
|
||||
logger.info(f"Task {task_id}: Splitting document into chunks")
|
||||
text_splitter = RecursiveCharacterTextSplitter(
|
||||
chunk_size=chunk_size,
|
||||
chunk_overlap=chunk_overlap
|
||||
)
|
||||
chunks = text_splitter.split_documents(documents)
|
||||
logger.info(f"Task {task_id}: Document split into {len(chunks)} chunks")
|
||||
|
||||
# 3. 创建向量存储
|
||||
logger.info(f"Task {task_id}: Initializing vector store")
|
||||
embeddings = EmbeddingsFactory.create()
|
||||
|
||||
vector_store = VectorStoreFactory.create(
|
||||
store_type=settings.VECTOR_STORE_TYPE,
|
||||
collection_name=f"kb_{kb_id}",
|
||||
embedding_function=embeddings,
|
||||
)
|
||||
|
||||
# 4. 将临时文件移动到永久目录
|
||||
permanent_path = f"kb_{kb_id}/{file_name}"
|
||||
try:
|
||||
logger.info(f"Task {task_id}: Moving file to permanent storage")
|
||||
# 复制到永久目录
|
||||
source = CopySource(settings.MINIO_BUCKET_NAME, temp_path)
|
||||
minio_client.copy_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=permanent_path,
|
||||
source=source
|
||||
)
|
||||
logger.info(f"Task {task_id}: File moved to permanent storage")
|
||||
|
||||
# 删除临时文件
|
||||
logger.info(f"Task {task_id}: Removing temporary file from MinIO")
|
||||
minio_client.remove_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=temp_path
|
||||
)
|
||||
logger.info(f"Task {task_id}: Temporary file removed")
|
||||
except MinioException as e:
|
||||
error_msg = f"Failed to move file to permanent storage: {str(e)}"
|
||||
logger.error(f"Task {task_id}: {error_msg}")
|
||||
raise Exception(error_msg)
|
||||
|
||||
# 5. 创建文档记录
|
||||
logger.info(f"Task {task_id}: Creating document record")
|
||||
document = Document(
|
||||
file_name=file_name,
|
||||
file_path=permanent_path,
|
||||
file_hash=task.document_upload.file_hash,
|
||||
file_size=task.document_upload.file_size,
|
||||
content_type=task.document_upload.content_type,
|
||||
knowledge_base_id=kb_id
|
||||
)
|
||||
db.add(document)
|
||||
db.flush()
|
||||
db.refresh(document)
|
||||
logger.info(f"Task {task_id}: Document record created with ID {document.id}")
|
||||
|
||||
# 6. 存储文档块
|
||||
logger.info(f"Task {task_id}: Storing document chunks")
|
||||
for i, chunk in enumerate(chunks):
|
||||
# 为每个 chunk 生成唯一的 ID
|
||||
chunk_id = hashlib.sha256(
|
||||
f"{kb_id}:{file_name}:{chunk.page_content}".encode()
|
||||
).hexdigest()
|
||||
|
||||
metadata = _build_enriched_chunk_metadata(
|
||||
source_metadata=chunk.metadata,
|
||||
chunk_id=chunk_id,
|
||||
file_name=file_name,
|
||||
file_path=permanent_path,
|
||||
kb_id=kb_id,
|
||||
document_id=document.id,
|
||||
chunk_index=i,
|
||||
chunk_text=chunk.page_content,
|
||||
)
|
||||
chunk.metadata = metadata
|
||||
|
||||
doc_chunk = DocumentChunk(
|
||||
id=chunk_id, # 添加 ID 字段
|
||||
document_id=document.id,
|
||||
kb_id=kb_id,
|
||||
file_name=file_name,
|
||||
chunk_metadata={
|
||||
"page_content": chunk.page_content,
|
||||
**metadata
|
||||
},
|
||||
hash=hashlib.sha256(
|
||||
(chunk.page_content + str(metadata)).encode()
|
||||
).hexdigest()
|
||||
)
|
||||
db.add(doc_chunk)
|
||||
if i > 0 and i % 100 == 0:
|
||||
logger.info(f"Task {task_id}: Stored {i} chunks")
|
||||
db.flush()
|
||||
|
||||
# 7. 添加到向量存储
|
||||
logger.info(f"Task {task_id}: Adding chunks to vector store")
|
||||
vector_chunks = [
|
||||
LangchainDocument(
|
||||
page_content=chunk.page_content,
|
||||
metadata=_sanitize_metadata_for_vector_store(chunk.metadata),
|
||||
)
|
||||
for chunk in chunks
|
||||
]
|
||||
vector_store.add_documents(vector_chunks)
|
||||
# 移除 persist() 调用，因为新版本不需要
|
||||
logger.info(f"Task {task_id}: Chunks added to vector store")
|
||||
|
||||
if settings.GRAPHRAG_ENABLED:
|
||||
try:
|
||||
from app.services.graph.graphrag_adapter import GraphRAGAdapter
|
||||
|
||||
logger.info(f"Task {task_id}: Starting GraphRAG ingestion")
|
||||
graph_adapter = GraphRAGAdapter()
|
||||
source_texts = [doc.page_content for doc in documents if doc.page_content.strip()]
|
||||
await graph_adapter.ingest_texts(kb_id, source_texts)
|
||||
logger.info(f"Task {task_id}: GraphRAG ingestion completed")
|
||||
except Exception as graph_exc:
|
||||
logger.error(f"Task {task_id}: GraphRAG ingestion failed: {graph_exc}")
|
||||
|
||||
# 8. 更新任务状态
|
||||
logger.info(f"Task {task_id}: Updating task status to completed")
|
||||
task.status = "completed"
|
||||
task.document_id = document.id # 更新为新创建的文档ID
|
||||
|
||||
# 9. 更新上传记录状态
|
||||
upload = task.document_upload # 直接通过关系获取
|
||||
if upload:
|
||||
logger.info(f"Task {task_id}: Updating upload record status to completed")
|
||||
upload.status = "completed"
|
||||
|
||||
db.commit()
|
||||
logger.info(f"Task {task_id}: Processing completed successfully")
|
||||
|
||||
finally:
|
||||
# 清理本地临时文件
|
||||
try:
|
||||
if os.path.exists(local_temp_path):
|
||||
logger.info(f"Task {task_id}: Cleaning up local temp file")
|
||||
os.remove(local_temp_path)
|
||||
logger.info(f"Task {task_id}: Local temp file cleaned up")
|
||||
except Exception as e:
|
||||
logger.warning(f"Task {task_id}: Failed to clean up local temp file: {str(e)}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Task {task_id}: Error processing document: {str(e)}")
|
||||
logger.error(f"Task {task_id}: Stack trace: {traceback.format_exc()}")
|
||||
db.rollback()
|
||||
|
||||
failed_task = db.query(ProcessingTask).get(task_id)
|
||||
if failed_task:
|
||||
failed_task.status = "failed"
|
||||
failed_task.error_message = str(e)
|
||||
if failed_task.document_upload:
|
||||
failed_task.document_upload.status = "failed"
|
||||
failed_task.document_upload.error_message = str(e)
|
||||
db.commit()
|
||||
|
||||
# 清理临时文件
|
||||
try:
|
||||
logger.info(f"Task {task_id}: Cleaning up temporary file after error")
|
||||
if minio_client is not None:
|
||||
minio_client.remove_object(
|
||||
bucket_name=settings.MINIO_BUCKET_NAME,
|
||||
object_name=temp_path
|
||||
)
|
||||
logger.info(f"Task {task_id}: Temporary file cleaned up after error")
|
||||
except:
|
||||
logger.warning(f"Task {task_id}: Failed to clean up temporary file after error")
|
||||
finally:
|
||||
# if we create the db session, we need to close it
|
||||
if should_close_db and db:
|
||||
db.close()
|
||||
@@ -0,0 +1,46 @@
|
||||
from app.core.config import settings
|
||||
from langchain_openai import OpenAIEmbeddings
|
||||
from langchain_ollama import OllamaEmbeddings
|
||||
# If you plan on adding other embeddings, import them here
|
||||
# from some_other_module import AnotherEmbeddingClass
|
||||
|
||||
|
||||
class EmbeddingsFactory:
|
||||
@staticmethod
|
||||
def create():
|
||||
"""
|
||||
Factory method to create an embeddings instance based on .env config.
|
||||
"""
|
||||
# Suppose your .env has a value like EMBEDDINGS_PROVIDER=openai
|
||||
embeddings_provider = settings.EMBEDDINGS_PROVIDER.lower()
|
||||
|
||||
if embeddings_provider == "openai":
|
||||
return OpenAIEmbeddings(
|
||||
openai_api_key=settings.OPENAI_API_KEY,
|
||||
openai_api_base=settings.OPENAI_API_BASE,
|
||||
model=settings.OPENAI_EMBEDDINGS_MODEL
|
||||
)
|
||||
elif embeddings_provider == "dashscope":
|
||||
return OpenAIEmbeddings(
|
||||
openai_api_key=settings.DASH_SCOPE_API_KEY,
|
||||
openai_api_base=settings.DASH_SCOPE_API_BASE,
|
||||
model=settings.DASH_SCOPE_EMBEDDINGS_MODEL,
|
||||
# DashScope OpenAI-compatible embedding expects string input,
|
||||
# while LangChain's len-safe path may send token ids.
|
||||
check_embedding_ctx_length=False,
|
||||
tiktoken_enabled=False,
|
||||
skip_empty=True,
|
||||
# DashScope embedding API supports at most 10 inputs per batch.
|
||||
chunk_size=10,
|
||||
)
|
||||
elif embeddings_provider == "ollama":
|
||||
return OllamaEmbeddings(
|
||||
model=settings.OLLAMA_EMBEDDINGS_MODEL,
|
||||
base_url=settings.OLLAMA_API_BASE
|
||||
)
|
||||
|
||||
# Extend with other providers:
|
||||
# elif embeddings_provider == "another_provider":
|
||||
# return AnotherEmbeddingClass(...)
|
||||
else:
|
||||
raise ValueError(f"Unsupported embeddings provider: {embeddings_provider}")
|
||||
116
rag-web-ui/backend/app/services/fusion_prompts.py
Normal file
116
rag-web-ui/backend/app/services/fusion_prompts.py
Normal file
@@ -0,0 +1,116 @@
|
||||
"""Fusion RAG prompts for aerospace Chinese QA."""
|
||||
|
||||
ROUTER_SYSTEM_PROMPT = """
|
||||
你是一个检索路由器。你的唯一任务是把用户请求分类到以下四类之一。
|
||||
|
||||
分类标签：
|
||||
A: 通用对话路
|
||||
- 适用：问候、寒暄、角色扮演、无须知识库支持的常识闲聊。
|
||||
- 特征：没有明确的专业实体约束，也不依赖当前知识库文档。
|
||||
|
||||
B: 混合检索路 (Hybrid RAG)
|
||||
- 适用：单实体事实查询、定义解释、时间/数值/指标问答。
|
||||
- 特征：问题通常可由少量文本片段直接回答，核心是“找准证据”。
|
||||
|
||||
C: 局部图检索路 (Graph Local Search)
|
||||
- 适用：实体关系、多跳因果、组件依赖、跨段落链式推理。
|
||||
- 特征：问题包含“谁影响谁/为什么/如何传导/依赖链”。
|
||||
|
||||
D: 全局图检索路 (Graph Global Search)
|
||||
- 适用：全局总结、趋势分析、跨系统比较、宏观评估。
|
||||
- 特征：问题面向整个语料或多个主题社区，不是单点事实。
|
||||
|
||||
判定规则（按优先级）：
|
||||
1. 若请求明确是问候、寒暄、开放闲聊，判 A。
|
||||
2. 若请求强调全局综述、趋势、横向比较，判 D。
|
||||
3. 若请求强调实体关系、影响路径、多跳推理，判 C。
|
||||
4. 其余知识查询默认判 B。
|
||||
|
||||
输出要求：
|
||||
- 只能输出 JSON，不要额外文本。
|
||||
- 格式必须是：
|
||||
{
|
||||
"intent": "A/B/C/D",
|
||||
"reason": "中文简要理由"
|
||||
}
|
||||
""".strip()
|
||||
|
||||
ROUTER_USER_PROMPT_TEMPLATE = """
|
||||
请基于以下用户问题进行路由分类。
|
||||
|
||||
历史对话（可选）：
|
||||
{chat_history}
|
||||
|
||||
用户问题：
|
||||
{query}
|
||||
""".strip()
|
||||
|
||||
GENERAL_CHAT_PROMPT_TEMPLATE = """
|
||||
你是中文航天问答助手。当前请求被路由为“通用对话路”。
|
||||
请直接回答用户问题，要求：
|
||||
- 简洁自然
|
||||
- 不要伪造具体文献或数据来源
|
||||
- 若涉及专业细节但无上下文支撑，请明确说明是一般性知识
|
||||
|
||||
用户问题：
|
||||
{query}
|
||||
""".strip()
|
||||
|
||||
HYBRID_RAG_PROMPT_TEMPLATE = """
|
||||
你是航天领域事实问答助手。你会收到按相关性排序的文本证据片段，请严格基于证据作答。
|
||||
|
||||
要求：
|
||||
1. 回答正文应自然连贯，不要使用“直接答案”“证据依据”等分节标题。
|
||||
2. 关键信息需要有可追溯引用，引用编号使用 [1]、[2] 等格式。
|
||||
3. 引用标号尽量集中放在回答末尾，不要在句中频繁插入。
|
||||
4. 不得编造未在证据中出现的事实、时间、参数、型号。
|
||||
5. 若证据不足，明确写：信息不足，缺少 xxx。
|
||||
6. 输出中文，术语严谨，避免冗长。
|
||||
|
||||
问题：
|
||||
{query}
|
||||
|
||||
证据片段：
|
||||
{context}
|
||||
""".strip()
|
||||
|
||||
GRAPH_LOCAL_PROMPT_TEMPLATE = """
|
||||
你是航天知识图谱推理助手。你将获得一个局部子图上下文（实体、关系、证据）。
|
||||
|
||||
要求：
|
||||
1. 输出结构固定为：
|
||||
- 结论
|
||||
- 推理链路
|
||||
- 证据映射
|
||||
- 不确定性
|
||||
2. 推理链路需按步骤编号（步骤1、步骤2...），明确“实体 -> 关系 -> 实体/结论”的链式过程。
|
||||
3. 若局部子图不完整，必须指出断点，不能臆造链路。
|
||||
4. 输出中文。
|
||||
|
||||
问题：
|
||||
{query}
|
||||
|
||||
局部子图上下文：
|
||||
{graph_context}
|
||||
""".strip()
|
||||
|
||||
GRAPH_GLOBAL_PROMPT_TEMPLATE = """
|
||||
你是航天领域全局分析助手。你将获得多个社区摘要，请进行跨社区综合研判。
|
||||
|
||||
要求：
|
||||
1. 输出结构固定为：
|
||||
- 总体结论
|
||||
- 跨社区共性
|
||||
- 关键差异
|
||||
- 趋势判断
|
||||
- 风险与建议
|
||||
2. 每条关键判断尽量给出对应社区编号。
|
||||
3. 仅依据输入摘要，证据不足时明确说明。
|
||||
4. 输出中文，适合技术管理层阅读。
|
||||
|
||||
问题：
|
||||
{query}
|
||||
|
||||
社区摘要：
|
||||
{community_context}
|
||||
""".strip()
|
||||
3
rag-web-ui/backend/app/services/graph/__init__.py
Normal file
3
rag-web-ui/backend/app/services/graph/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from app.services.graph.graphrag_adapter import GraphRAGAdapter
|
||||
|
||||
__all__ = ["GraphRAGAdapter"]
|
||||
183
rag-web-ui/backend/app/services/graph/graphrag_adapter.py
Normal file
183
rag-web-ui/backend/app/services/graph/graphrag_adapter.py
Normal file
@@ -0,0 +1,183 @@
|
||||
import asyncio
|
||||
import importlib
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.embedding.embedding_factory import EmbeddingsFactory
|
||||
from app.services.llm.llm_factory import LLMFactory
|
||||
|
||||
|
||||
class GraphRAGAdapter:
|
||||
_instance_lock = asyncio.Lock()
|
||||
|
||||
def __init__(self):
|
||||
self._graphrag_instances: Dict[int, Any] = {}
|
||||
self._kb_locks: Dict[int, asyncio.Lock] = {}
|
||||
self._embedding_model = EmbeddingsFactory.create()
|
||||
self._llm_model = LLMFactory.create(streaming=False)
|
||||
self._symbols = self._load_symbols()
|
||||
|
||||
def _load_symbols(self) -> Dict[str, Any]:
|
||||
module = importlib.import_module("nano_graphrag")
|
||||
|
||||
storage_module = importlib.import_module("nano_graphrag._storage")
|
||||
utils_module = importlib.import_module("nano_graphrag._utils")
|
||||
|
||||
return {
|
||||
"GraphRAG": module.GraphRAG,
|
||||
"QueryParam": module.QueryParam,
|
||||
"Neo4jStorage": getattr(storage_module, "Neo4jStorage"),
|
||||
"NetworkXStorage": getattr(storage_module, "NetworkXStorage"),
|
||||
"EmbeddingFunc": getattr(utils_module, "EmbeddingFunc"),
|
||||
}
|
||||
|
||||
def _get_kb_lock(self, kb_id: int) -> asyncio.Lock:
|
||||
if kb_id not in self._kb_locks:
|
||||
self._kb_locks[kb_id] = asyncio.Lock()
|
||||
return self._kb_locks[kb_id]
|
||||
|
||||
async def _llm_complete(self, prompt: str, system_prompt: Optional[str] = None, history_messages: Optional[List[Any]] = None, **kwargs: Any) -> str:
|
||||
history_messages = history_messages or []
|
||||
|
||||
history_lines: List[str] = []
|
||||
for item in history_messages:
|
||||
if isinstance(item, dict):
|
||||
role = str(item.get("role", "user"))
|
||||
content = item.get("content", "")
|
||||
if isinstance(content, list):
|
||||
joined = " ".join(str(part.get("text", "")) for part in content if isinstance(part, dict))
|
||||
history_lines.append(f"{role}: {joined}")
|
||||
else:
|
||||
history_lines.append(f"{role}: {content}")
|
||||
else:
|
||||
history_lines.append(str(item))
|
||||
|
||||
full_prompt = "\n\n".join(
|
||||
part
|
||||
for part in [
|
||||
f"系统提示: {system_prompt}" if system_prompt else "",
|
||||
"历史对话:\n" + "\n".join(history_lines) if history_lines else "",
|
||||
"用户输入:\n" + prompt,
|
||||
]
|
||||
if part
|
||||
)
|
||||
|
||||
model = self._llm_model
|
||||
max_tokens = kwargs.get("max_tokens")
|
||||
if max_tokens is not None:
|
||||
try:
|
||||
model = model.bind(max_tokens=max_tokens)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
response = await model.ainvoke(full_prompt)
|
||||
content = getattr(response, "content", response)
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
return str(content)
|
||||
|
||||
async def _embedding_call(self, texts: List[str]) -> np.ndarray:
|
||||
vectors = await asyncio.to_thread(self._embedding_model.embed_documents, texts)
|
||||
return np.array(vectors)
|
||||
|
||||
async def _get_or_create(self, kb_id: int) -> Any:
|
||||
if kb_id in self._graphrag_instances:
|
||||
return self._graphrag_instances[kb_id]
|
||||
|
||||
async with GraphRAGAdapter._instance_lock:
|
||||
if kb_id in self._graphrag_instances:
|
||||
return self._graphrag_instances[kb_id]
|
||||
|
||||
GraphRAG = self._symbols["GraphRAG"]
|
||||
EmbeddingFunc = self._symbols["EmbeddingFunc"]
|
||||
|
||||
embedding_func = EmbeddingFunc(
|
||||
embedding_dim=settings.GRAPHRAG_EMBEDDING_DIM,
|
||||
max_token_size=settings.GRAPHRAG_EMBEDDING_MAX_TOKEN_SIZE,
|
||||
func=self._embedding_call,
|
||||
)
|
||||
|
||||
graph_storage_cls = self._symbols["NetworkXStorage"]
|
||||
addon_params: Dict[str, Any] = {}
|
||||
if settings.GRAPHRAG_GRAPH_STORAGE.lower() == "neo4j":
|
||||
graph_storage_cls = self._symbols["Neo4jStorage"]
|
||||
addon_params = {
|
||||
"neo4j_url": settings.NEO4J_URL,
|
||||
"neo4j_auth": (settings.NEO4J_USERNAME, settings.NEO4J_PASSWORD),
|
||||
}
|
||||
|
||||
working_dir = str(Path(settings.GRAPHRAG_WORKING_DIR) / f"kb_{kb_id}")
|
||||
|
||||
rag = GraphRAG(
|
||||
working_dir=working_dir,
|
||||
enable_local=True,
|
||||
enable_naive_rag=True,
|
||||
graph_storage_cls=graph_storage_cls,
|
||||
addon_params=addon_params,
|
||||
embedding_func=embedding_func,
|
||||
best_model_func=self._llm_complete,
|
||||
cheap_model_func=self._llm_complete,
|
||||
entity_extract_max_gleaning=settings.GRAPHRAG_ENTITY_EXTRACT_MAX_GLEANING,
|
||||
)
|
||||
self._graphrag_instances[kb_id] = rag
|
||||
return rag
|
||||
|
||||
async def ingest_texts(self, kb_id: int, texts: List[str]) -> None:
|
||||
cleaned = [text.strip() for text in texts if text and text.strip()]
|
||||
if not cleaned:
|
||||
return
|
||||
|
||||
rag = await self._get_or_create(kb_id)
|
||||
lock = self._get_kb_lock(kb_id)
|
||||
async with lock:
|
||||
await rag.ainsert(cleaned)
|
||||
|
||||
async def local_context(self, kb_id: int, query: str, *, top_k: int = 20, level: int = 2) -> str:
|
||||
rag = await self._get_or_create(kb_id)
|
||||
QueryParam = self._symbols["QueryParam"]
|
||||
param = QueryParam(
|
||||
mode="local",
|
||||
top_k=top_k,
|
||||
level=level,
|
||||
only_need_context=True,
|
||||
)
|
||||
return await rag.aquery(query, param)
|
||||
|
||||
async def global_context(self, kb_id: int, query: str, *, level: int = 2) -> str:
|
||||
rag = await self._get_or_create(kb_id)
|
||||
QueryParam = self._symbols["QueryParam"]
|
||||
param = QueryParam(
|
||||
mode="global",
|
||||
level=level,
|
||||
only_need_context=True,
|
||||
)
|
||||
return await rag.aquery(query, param)
|
||||
|
||||
async def local_context_multi(self, kb_ids: List[int], query: str, *, top_k: int = 20, level: int = 2) -> Tuple[str, List[int]]:
|
||||
contexts: List[str] = []
|
||||
used_kb_ids: List[int] = []
|
||||
for kb_id in kb_ids:
|
||||
try:
|
||||
ctx = await self.local_context(kb_id, query, top_k=top_k, level=level)
|
||||
if ctx:
|
||||
contexts.append(f"[KB:{kb_id}]\n{ctx}")
|
||||
used_kb_ids.append(kb_id)
|
||||
except Exception:
|
||||
continue
|
||||
return "\n\n".join(contexts), used_kb_ids
|
||||
|
||||
async def global_context_multi(self, kb_ids: List[int], query: str, *, level: int = 2) -> Tuple[str, List[int]]:
|
||||
contexts: List[str] = []
|
||||
used_kb_ids: List[int] = []
|
||||
for kb_id in kb_ids:
|
||||
try:
|
||||
ctx = await self.global_context(kb_id, query, level=level)
|
||||
if ctx:
|
||||
contexts.append(f"[KB:{kb_id}]\n{ctx}")
|
||||
used_kb_ids.append(kb_id)
|
||||
except Exception:
|
||||
continue
|
||||
return "\n\n".join(contexts), used_kb_ids
|
||||
85
rag-web-ui/backend/app/services/hybrid_retriever.py
Normal file
85
rag-web-ui/backend/app/services/hybrid_retriever.py
Normal file
@@ -0,0 +1,85 @@
|
||||
import re
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from app.services.vector_store.base import BaseVectorStore
|
||||
|
||||
|
||||
def _tokenize_for_keyword_score(text: str) -> List[str]:
|
||||
"""Simple multilingual tokenizer for lexical matching without extra dependencies."""
|
||||
tokens = re.findall(r"[A-Za-z0-9_]+|[\u4e00-\u9fff]", text.lower())
|
||||
return [token for token in tokens if token.strip()]
|
||||
|
||||
|
||||
def _keyword_score(query: str, doc_text: str) -> float:
|
||||
query_terms = set(_tokenize_for_keyword_score(query))
|
||||
doc_terms = set(_tokenize_for_keyword_score(doc_text))
|
||||
|
||||
if not query_terms or not doc_terms:
|
||||
return 0.0
|
||||
|
||||
overlap = len(query_terms.intersection(doc_terms))
|
||||
return overlap / max(1, len(query_terms))
|
||||
|
||||
|
||||
def hybrid_search(
|
||||
vector_store: BaseVectorStore,
|
||||
query: str,
|
||||
top_k: int = 6,
|
||||
fetch_k: int = 20,
|
||||
alpha: float = 0.65,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Hybrid retrieval via vector candidate generation + lexical reranking.
|
||||
|
||||
score = alpha * vector_rank_score + (1 - alpha) * keyword_score
|
||||
"""
|
||||
raw_results = vector_store.similarity_search_with_score(query, k=fetch_k)
|
||||
if not raw_results:
|
||||
return []
|
||||
|
||||
ranked: List[Dict[str, Any]] = []
|
||||
total = len(raw_results)
|
||||
|
||||
for index, item in enumerate(raw_results):
|
||||
if not isinstance(item, (tuple, list)) or len(item) < 1:
|
||||
continue
|
||||
|
||||
doc = item[0]
|
||||
if not hasattr(doc, "page_content"):
|
||||
continue
|
||||
|
||||
rank_score = 1.0 - (index / max(1, total))
|
||||
lexical_score = _keyword_score(query, doc.page_content)
|
||||
final_score = alpha * rank_score + (1.0 - alpha) * lexical_score
|
||||
|
||||
ranked.append(
|
||||
{
|
||||
"document": doc,
|
||||
"vector_rank_score": round(rank_score, 6),
|
||||
"keyword_score": round(lexical_score, 6),
|
||||
"final_score": round(final_score, 6),
|
||||
}
|
||||
)
|
||||
|
||||
ranked.sort(key=lambda row: row["final_score"], reverse=True)
|
||||
return ranked[:top_k]
|
||||
|
||||
|
||||
def format_hybrid_context(rows: List[Dict[str, Any]]) -> str:
|
||||
parts: List[str] = []
|
||||
|
||||
for i, row in enumerate(rows, start=1):
|
||||
doc = row["document"]
|
||||
metadata = doc.metadata or {}
|
||||
source = metadata.get("source") or metadata.get("file_name") or "unknown"
|
||||
chunk_id = metadata.get("chunk_id") or "unknown"
|
||||
|
||||
parts.append(
|
||||
(
|
||||
f"[{i}] source={source}, chunk_id={chunk_id}, "
|
||||
f"score={row['final_score']}\n"
|
||||
f"{doc.page_content.strip()}"
|
||||
)
|
||||
)
|
||||
|
||||
return "\n\n".join(parts)
|
||||
120
rag-web-ui/backend/app/services/intent_router.py
Normal file
120
rag-web-ui/backend/app/services/intent_router.py
Normal file
@@ -0,0 +1,120 @@
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from app.services.fusion_prompts import (
|
||||
ROUTER_SYSTEM_PROMPT,
|
||||
ROUTER_USER_PROMPT_TEMPLATE,
|
||||
)
|
||||
|
||||
VALID_INTENTS = {"A", "B", "C", "D"}
|
||||
|
||||
|
||||
def _extract_json_object(raw_text: str) -> Dict[str, str]:
|
||||
"""Extract and parse the first JSON object from model output."""
|
||||
cleaned = raw_text.strip()
|
||||
cleaned = cleaned.replace("```json", "").replace("```", "").strip()
|
||||
|
||||
match = re.search(r"\{[\s\S]*\}", cleaned)
|
||||
if not match:
|
||||
raise ValueError("No JSON object found in router output")
|
||||
|
||||
data = json.loads(match.group(0))
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("Router output JSON is not an object")
|
||||
|
||||
intent = str(data.get("intent", "")).strip().upper()
|
||||
reason = str(data.get("reason", "")).strip()
|
||||
if intent not in VALID_INTENTS:
|
||||
raise ValueError(f"Invalid intent: {intent}")
|
||||
|
||||
if not reason:
|
||||
reason = "模型未提供理由，已按规则兜底。"
|
||||
|
||||
return {"intent": intent, "reason": reason}
|
||||
|
||||
|
||||
def _build_history_text(messages: dict, max_turns: int = 6) -> str:
|
||||
if not isinstance(messages, dict):
|
||||
return ""
|
||||
|
||||
history = messages.get("messages", [])
|
||||
if not isinstance(history, list):
|
||||
return ""
|
||||
|
||||
tail = history[-max_turns:]
|
||||
rows: List[str] = []
|
||||
for msg in tail:
|
||||
role = str(msg.get("role", "unknown")).strip()
|
||||
content = str(msg.get("content", "")).strip().replace("\n", " ")
|
||||
if content:
|
||||
rows.append(f"{role}: {content}")
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def _heuristic_route(query: str) -> Dict[str, str]:
|
||||
text = query.strip().lower()
|
||||
|
||||
general_chat_patterns = [
|
||||
"你好",
|
||||
"您好",
|
||||
"在吗",
|
||||
"谢谢",
|
||||
"早上好",
|
||||
"晚上好",
|
||||
"你是谁",
|
||||
"讲个笑话",
|
||||
]
|
||||
global_patterns = [
|
||||
"总结",
|
||||
"综述",
|
||||
"整体",
|
||||
"全局",
|
||||
"趋势",
|
||||
"对比",
|
||||
"比较",
|
||||
"宏观",
|
||||
"共性",
|
||||
"差异",
|
||||
]
|
||||
local_graph_patterns = [
|
||||
"关系",
|
||||
"依赖",
|
||||
"影响",
|
||||
"导致",
|
||||
"原因",
|
||||
"链路",
|
||||
"多跳",
|
||||
"传导",
|
||||
"耦合",
|
||||
"约束",
|
||||
]
|
||||
|
||||
if any(token in text for token in general_chat_patterns):
|
||||
return {"intent": "A", "reason": "命中通用对话关键词，且不依赖知识库检索。"}
|
||||
|
||||
if any(token in text for token in global_patterns):
|
||||
return {"intent": "D", "reason": "问题指向全局总结或跨主题趋势分析。"}
|
||||
|
||||
if any(token in text for token in local_graph_patterns):
|
||||
return {"intent": "C", "reason": "问题强调实体关系与链式推理。"}
|
||||
|
||||
return {"intent": "B", "reason": "默认归入事实查询，适合混合检索链路。"}
|
||||
|
||||
|
||||
async def route_intent(llm: Any, query: str, messages: dict) -> Dict[str, str]:
|
||||
"""Route user query to A/B/C/D with LLM-first and heuristic fallback."""
|
||||
history_text = _build_history_text(messages)
|
||||
user_prompt = ROUTER_USER_PROMPT_TEMPLATE.format(
|
||||
chat_history=history_text or "无",
|
||||
query=query,
|
||||
)
|
||||
|
||||
try:
|
||||
full_prompt = f"{ROUTER_SYSTEM_PROMPT}\n\n{user_prompt}"
|
||||
model_resp = await llm.ainvoke(full_prompt)
|
||||
content = getattr(model_resp, "content", model_resp)
|
||||
raw_text = content if isinstance(content, str) else str(content)
|
||||
return _extract_json_object(raw_text)
|
||||
except Exception:
|
||||
return _heuristic_route(query)
|
||||
57
rag-web-ui/backend/app/services/llm/llm_factory.py
Normal file
57
rag-web-ui/backend/app/services/llm/llm_factory.py
Normal file
@@ -0,0 +1,57 @@
|
||||
from typing import Optional
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_openai import ChatOpenAI
|
||||
from langchain_deepseek import ChatDeepSeek
|
||||
from langchain_ollama import OllamaLLM
|
||||
from app.core.config import settings
|
||||
|
||||
class LLMFactory:
|
||||
@staticmethod
|
||||
def create(
|
||||
provider: Optional[str] = None,
|
||||
temperature: float = 0,
|
||||
streaming: bool = True,
|
||||
) -> BaseChatModel:
|
||||
"""
|
||||
Create a LLM instance based on the provider
|
||||
"""
|
||||
# If no provider specified, use the one from settings
|
||||
provider = provider or settings.CHAT_PROVIDER
|
||||
|
||||
if provider.lower() == "openai":
|
||||
return ChatOpenAI(
|
||||
temperature=temperature,
|
||||
streaming=streaming,
|
||||
model=settings.OPENAI_MODEL,
|
||||
openai_api_key=settings.OPENAI_API_KEY,
|
||||
openai_api_base=settings.OPENAI_API_BASE
|
||||
)
|
||||
elif provider.lower() == "deepseek":
|
||||
return ChatDeepSeek(
|
||||
temperature=temperature,
|
||||
streaming=streaming,
|
||||
model=settings.DEEPSEEK_MODEL,
|
||||
api_key=settings.DEEPSEEK_API_KEY,
|
||||
api_base=settings.DEEPSEEK_API_BASE
|
||||
)
|
||||
elif provider.lower() == "dashscope":
|
||||
return ChatOpenAI(
|
||||
temperature=temperature,
|
||||
streaming=streaming,
|
||||
model=settings.DASH_SCOPE_CHAT_MODEL,
|
||||
openai_api_key=settings.DASH_SCOPE_API_KEY,
|
||||
openai_api_base=settings.DASH_SCOPE_API_BASE,
|
||||
)
|
||||
elif provider.lower() == "ollama":
|
||||
# Initialize Ollama model
|
||||
return OllamaLLM(
|
||||
model=settings.OLLAMA_MODEL,
|
||||
base_url=settings.OLLAMA_API_BASE,
|
||||
temperature=temperature,
|
||||
streaming=streaming
|
||||
)
|
||||
# Add more providers here as needed
|
||||
# elif provider.lower() == "anthropic":
|
||||
# return ChatAnthropic(...)
|
||||
else:
|
||||
raise ValueError(f"Unsupported LLM provider: {provider}")
|
||||
3
rag-web-ui/backend/app/services/reranker/__init__.py
Normal file
3
rag-web-ui/backend/app/services/reranker/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from app.services.reranker.external_api import ExternalRerankerClient
|
||||
|
||||
__all__ = ["ExternalRerankerClient"]
|
||||
164
rag-web-ui/backend/app/services/reranker/external_api.py
Normal file
164
rag-web-ui/backend/app/services/reranker/external_api.py
Normal file
@@ -0,0 +1,164 @@
|
||||
import asyncio
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib import request
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExternalRerankerClient:
|
||||
api_url: str
|
||||
api_key: str = ""
|
||||
model: str = ""
|
||||
timeout_seconds: float = 8.0
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return bool(self.api_url)
|
||||
|
||||
@property
|
||||
def is_dashscope_rerank(self) -> bool:
|
||||
return "dashscope.aliyuncs.com" in self.api_url and "/services/rerank/" in self.api_url
|
||||
|
||||
async def rerank(
|
||||
self,
|
||||
*,
|
||||
query: str,
|
||||
documents: List[str],
|
||||
top_n: Optional[int] = None,
|
||||
metadata: Optional[List[Dict[str, Any]]] = None,
|
||||
) -> Optional[List[float]]:
|
||||
if not self.enabled:
|
||||
return None
|
||||
if not documents:
|
||||
return []
|
||||
|
||||
payload = self._build_payload(
|
||||
query=query,
|
||||
documents=documents,
|
||||
top_n=top_n or len(documents),
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
try:
|
||||
response = await asyncio.to_thread(self._post_json, payload)
|
||||
scores = self._parse_scores(response, len(documents))
|
||||
return scores
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _post_json(self, payload: Dict[str, Any]) -> Dict[str, Any]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if self.api_key:
|
||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
||||
|
||||
req = request.Request(
|
||||
self.api_url,
|
||||
data=json.dumps(payload).encode("utf-8"),
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
with request.urlopen(req, timeout=self.timeout_seconds) as resp:
|
||||
body = resp.read().decode("utf-8")
|
||||
return json.loads(body)
|
||||
|
||||
def _build_payload(
|
||||
self,
|
||||
*,
|
||||
query: str,
|
||||
documents: List[str],
|
||||
top_n: int,
|
||||
metadata: Optional[List[Dict[str, Any]]],
|
||||
) -> Dict[str, Any]:
|
||||
if self.is_dashscope_rerank:
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"input": {
|
||||
"query": query,
|
||||
"documents": documents,
|
||||
},
|
||||
"parameters": {
|
||||
"return_documents": True,
|
||||
"top_n": top_n,
|
||||
},
|
||||
}
|
||||
if metadata:
|
||||
payload["metadata"] = metadata
|
||||
return payload
|
||||
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"query": query,
|
||||
"documents": documents,
|
||||
"top_n": top_n,
|
||||
}
|
||||
if metadata:
|
||||
payload["metadata"] = metadata
|
||||
return payload
|
||||
|
||||
def _parse_scores(self, response: Dict[str, Any], expected_len: int) -> List[float]:
|
||||
# DashScope format:
|
||||
# {"output": {"results": [{"index": 0, "relevance_score": 0.98}, ...]}}
|
||||
output_block = response.get("output")
|
||||
if isinstance(output_block, dict) and isinstance(output_block.get("results"), list):
|
||||
raw_results = output_block["results"]
|
||||
scores = [0.0] * expected_len
|
||||
for item in raw_results:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
idx = item.get("index")
|
||||
score = item.get("relevance_score", item.get("score", 0.0))
|
||||
if isinstance(idx, int) and 0 <= idx < expected_len:
|
||||
try:
|
||||
scores[idx] = float(score)
|
||||
except Exception:
|
||||
scores[idx] = 0.0
|
||||
return scores
|
||||
|
||||
# Common response format #1:
|
||||
# {"results": [{"index": 0, "relevance_score": 0.98}, ...]}
|
||||
if isinstance(response.get("results"), list):
|
||||
raw_results = response["results"]
|
||||
scores = [0.0] * expected_len
|
||||
for item in raw_results:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
idx = item.get("index")
|
||||
score = item.get("relevance_score", item.get("score", 0.0))
|
||||
if isinstance(idx, int) and 0 <= idx < expected_len:
|
||||
try:
|
||||
scores[idx] = float(score)
|
||||
except Exception:
|
||||
scores[idx] = 0.0
|
||||
return scores
|
||||
|
||||
# Common response format #2:
|
||||
# {"scores": [0.9, 0.1, ...]}
|
||||
if isinstance(response.get("scores"), list):
|
||||
values = response["scores"]
|
||||
scores: List[float] = []
|
||||
for i in range(expected_len):
|
||||
try:
|
||||
scores.append(float(values[i]))
|
||||
except Exception:
|
||||
scores.append(0.0)
|
||||
return scores
|
||||
|
||||
# Common response format #3:
|
||||
# {"data": [{"index": 0, "score": 0.88}, ...]}
|
||||
if isinstance(response.get("data"), list):
|
||||
raw_results = response["data"]
|
||||
scores = [0.0] * expected_len
|
||||
for item in raw_results:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
idx = item.get("index")
|
||||
score = item.get("score", item.get("relevance_score", 0.0))
|
||||
if isinstance(idx, int) and 0 <= idx < expected_len:
|
||||
try:
|
||||
scores[idx] = float(score)
|
||||
except Exception:
|
||||
scores[idx] = 0.0
|
||||
return scores
|
||||
|
||||
return [0.0] * expected_len
|
||||
3
rag-web-ui/backend/app/services/retrieval/__init__.py
Normal file
3
rag-web-ui/backend/app/services/retrieval/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from app.services.retrieval.multi_kb_retriever import MultiKBRetriever, format_retrieval_context
|
||||
|
||||
__all__ = ["MultiKBRetriever", "format_retrieval_context"]
|
||||
131
rag-web-ui/backend/app/services/retrieval/multi_kb_retriever.py
Normal file
131
rag-web-ui/backend/app/services/retrieval/multi_kb_retriever.py
Normal file
@@ -0,0 +1,131 @@
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from app.services.reranker.external_api import ExternalRerankerClient
|
||||
|
||||
|
||||
def _tokenize(text: str) -> List[str]:
|
||||
tokens = re.findall(r"[A-Za-z0-9_]+|[\u4e00-\u9fff]", text.lower())
|
||||
return [token for token in tokens if token.strip()]
|
||||
|
||||
|
||||
def _keyword_score(query: str, text: str) -> float:
|
||||
query_terms = set(_tokenize(query))
|
||||
text_terms = set(_tokenize(text))
|
||||
if not query_terms or not text_terms:
|
||||
return 0.0
|
||||
overlap = len(query_terms.intersection(text_terms))
|
||||
return overlap / max(1, len(query_terms))
|
||||
|
||||
|
||||
def format_retrieval_context(rows: List[Dict[str, Any]]) -> str:
|
||||
blocks: List[str] = []
|
||||
for i, row in enumerate(rows, start=1):
|
||||
doc = row["document"]
|
||||
metadata = doc.metadata or {}
|
||||
blocks.append(
|
||||
(
|
||||
f"[{i}] kb_id={row.get('kb_id')}, source={metadata.get('source') or metadata.get('file_name') or 'unknown'}, "
|
||||
f"chunk_id={metadata.get('chunk_id') or 'unknown'}, score={row.get('final_score', 0):.6f}\n"
|
||||
f"{doc.page_content.strip()}"
|
||||
)
|
||||
)
|
||||
return "\n\n".join(blocks)
|
||||
|
||||
|
||||
class MultiKBRetriever:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
reranker_client: Optional[ExternalRerankerClient] = None,
|
||||
reranker_weight: float = 0.75,
|
||||
vector_weight: float = 0.2,
|
||||
keyword_weight: float = 0.05,
|
||||
):
|
||||
self.reranker_client = reranker_client
|
||||
self.reranker_weight = reranker_weight
|
||||
self.vector_weight = vector_weight
|
||||
self.keyword_weight = keyword_weight
|
||||
|
||||
async def retrieve(
|
||||
self,
|
||||
*,
|
||||
query: str,
|
||||
kb_vector_stores: List[Dict[str, Any]],
|
||||
fetch_k_per_kb: int = 12,
|
||||
top_k: int = 12,
|
||||
) -> List[Dict[str, Any]]:
|
||||
candidates: List[Dict[str, Any]] = []
|
||||
|
||||
for kb_store in kb_vector_stores:
|
||||
kb_id = kb_store["kb_id"]
|
||||
vector_store = kb_store["store"]
|
||||
raw = vector_store.similarity_search_with_score(query, k=fetch_k_per_kb)
|
||||
total = len(raw)
|
||||
|
||||
for index, item in enumerate(raw):
|
||||
if not isinstance(item, (tuple, list)) or not item:
|
||||
continue
|
||||
|
||||
doc = item[0]
|
||||
if not hasattr(doc, "page_content"):
|
||||
continue
|
||||
|
||||
metadata = doc.metadata or {}
|
||||
rank_score = 1.0 - (index / max(1, total))
|
||||
lexical_score = _keyword_score(query, doc.page_content)
|
||||
|
||||
candidates.append(
|
||||
{
|
||||
"kb_id": kb_id,
|
||||
"document": doc,
|
||||
"chunk_key": f"{kb_id}:{metadata.get('chunk_id', index)}",
|
||||
"vector_rank_score": round(rank_score, 6),
|
||||
"keyword_score": round(lexical_score, 6),
|
||||
}
|
||||
)
|
||||
|
||||
if not candidates:
|
||||
return []
|
||||
|
||||
# Dedupe by KB + chunk id to avoid repeated chunks from same collection.
|
||||
unique_map: Dict[str, Dict[str, Any]] = {}
|
||||
for row in candidates:
|
||||
key = row["chunk_key"]
|
||||
existing = unique_map.get(key)
|
||||
if existing is None:
|
||||
unique_map[key] = row
|
||||
continue
|
||||
if row["vector_rank_score"] > existing["vector_rank_score"]:
|
||||
unique_map[key] = row
|
||||
|
||||
merged = list(unique_map.values())
|
||||
merged.sort(key=lambda x: x["vector_rank_score"], reverse=True)
|
||||
|
||||
reranker_scores: Optional[List[float]] = None
|
||||
if self.reranker_client is not None and self.reranker_client.enabled:
|
||||
reranker_scores = await self.reranker_client.rerank(
|
||||
query=query,
|
||||
documents=[row["document"].page_content for row in merged],
|
||||
top_n=min(top_k, len(merged)),
|
||||
metadata=[{"kb_id": row["kb_id"]} for row in merged],
|
||||
)
|
||||
|
||||
for idx, row in enumerate(merged):
|
||||
base_score = (
|
||||
self.vector_weight * row["vector_rank_score"]
|
||||
+ self.keyword_weight * row["keyword_score"]
|
||||
)
|
||||
|
||||
if reranker_scores is not None:
|
||||
rerank_value = float(reranker_scores[idx])
|
||||
final_score = self.reranker_weight * rerank_value + (1 - self.reranker_weight) * base_score
|
||||
row["reranker_score"] = round(rerank_value, 6)
|
||||
else:
|
||||
final_score = base_score
|
||||
row["reranker_score"] = None
|
||||
|
||||
row["final_score"] = round(final_score, 6)
|
||||
|
||||
merged.sort(key=lambda x: x["final_score"], reverse=True)
|
||||
return merged[:top_k]
|
||||
187
rag-web-ui/backend/app/services/srs_job_service.py
Normal file
187
rag-web-ui/backend/app/services/srs_job_service.py
Normal file
@@ -0,0 +1,187 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import SessionLocal
|
||||
from app.models.tooling import SRSExtraction, SRSRequirement, ToolJob
|
||||
from app.tools.srs_reqs_qwen import get_srs_tool
|
||||
|
||||
|
||||
def run_srs_job(job_id: int) -> None:
|
||||
db = SessionLocal()
|
||||
try:
|
||||
job = db.query(ToolJob).filter(ToolJob.id == job_id).first()
|
||||
if not job:
|
||||
return
|
||||
|
||||
job.status = "processing"
|
||||
job.started_at = datetime.utcnow()
|
||||
job.error_message = None
|
||||
db.commit()
|
||||
|
||||
payload = get_srs_tool().run(job.input_file_path)
|
||||
|
||||
extraction = SRSExtraction(
|
||||
job_id=job.id,
|
||||
document_name=payload["document_name"],
|
||||
document_title=payload.get("document_title") or payload["document_name"],
|
||||
generated_at=_parse_generated_at(payload.get("generated_at")),
|
||||
total_requirements=len(payload.get("requirements", [])),
|
||||
statistics=payload.get("statistics", {}),
|
||||
raw_output=payload.get("raw_output", {}),
|
||||
)
|
||||
db.add(extraction)
|
||||
db.flush()
|
||||
|
||||
for item in payload.get("requirements", []):
|
||||
requirement = SRSRequirement(
|
||||
extraction_id=extraction.id,
|
||||
requirement_uid=item["id"],
|
||||
title=item.get("title") or item["id"],
|
||||
description=item.get("description") or "",
|
||||
priority=item.get("priority") or "中",
|
||||
acceptance_criteria=item.get("acceptance_criteria") or ["待补充验收标准"],
|
||||
source_field=item.get("source_field") or "文档解析",
|
||||
section_number=item.get("section_number"),
|
||||
section_title=item.get("section_title"),
|
||||
requirement_type=item.get("requirement_type"),
|
||||
sort_order=int(item.get("sort_order") or 0),
|
||||
)
|
||||
db.add(requirement)
|
||||
|
||||
job.status = "completed"
|
||||
job.completed_at = datetime.utcnow()
|
||||
job.output_summary = {
|
||||
"total_requirements": extraction.total_requirements,
|
||||
"document_name": extraction.document_name,
|
||||
}
|
||||
db.commit()
|
||||
except Exception as exc:
|
||||
db.rollback()
|
||||
_mark_job_failed(job_id=job_id, error_message=str(exc))
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def _mark_job_failed(job_id: int, error_message: str) -> None:
|
||||
db = SessionLocal()
|
||||
try:
|
||||
job = db.query(ToolJob).filter(ToolJob.id == job_id).first()
|
||||
if not job:
|
||||
return
|
||||
job.status = "failed"
|
||||
job.completed_at = datetime.utcnow()
|
||||
job.error_message = error_message[:2000]
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def _parse_generated_at(value: Any) -> datetime:
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return datetime.fromisoformat(value)
|
||||
except ValueError:
|
||||
return datetime.utcnow()
|
||||
return datetime.utcnow()
|
||||
|
||||
|
||||
def ensure_upload_path(job_id: int, file_name: str) -> Path:
|
||||
target_dir = Path("uploads") / "srs_jobs" / str(job_id)
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
return target_dir / file_name
|
||||
|
||||
|
||||
def build_result_response(job: ToolJob, extraction: SRSExtraction) -> Dict[str, Any]:
|
||||
requirements: List[Dict[str, Any]] = []
|
||||
for item in extraction.requirements:
|
||||
requirements.append(
|
||||
{
|
||||
"id": item.requirement_uid,
|
||||
"title": item.title,
|
||||
"description": item.description,
|
||||
"priority": item.priority,
|
||||
"acceptanceCriteria": item.acceptance_criteria or [],
|
||||
"sourceField": item.source_field,
|
||||
"sectionNumber": item.section_number,
|
||||
"sectionTitle": item.section_title,
|
||||
"requirementType": item.requirement_type,
|
||||
"sortOrder": item.sort_order,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"jobId": job.id,
|
||||
"documentName": extraction.document_name,
|
||||
"generatedAt": extraction.generated_at.isoformat(),
|
||||
"statistics": extraction.statistics or {},
|
||||
"requirements": requirements,
|
||||
}
|
||||
|
||||
|
||||
def replace_requirements(db: Session, extraction: SRSExtraction, updates: List[Dict[str, Any]]) -> None:
|
||||
existing = {
|
||||
req.requirement_uid: req
|
||||
for req in db.query(SRSRequirement)
|
||||
.filter(SRSRequirement.extraction_id == extraction.id)
|
||||
.all()
|
||||
}
|
||||
seen_ids = set()
|
||||
|
||||
for index, item in enumerate(updates):
|
||||
uid = item["id"]
|
||||
seen_ids.add(uid)
|
||||
req = existing.get(uid)
|
||||
if req is None:
|
||||
req = SRSRequirement(
|
||||
extraction_id=extraction.id,
|
||||
requirement_uid=uid,
|
||||
title=item.get("title") or uid,
|
||||
description=item.get("description") if item.get("description") is not None else "",
|
||||
priority=item.get("priority") or "中",
|
||||
acceptance_criteria=item.get("acceptanceCriteria") or ["待补充验收标准"],
|
||||
source_field=item.get("sourceField") or "文档解析",
|
||||
section_number=item.get("sectionNumber"),
|
||||
section_title=item.get("sectionTitle"),
|
||||
requirement_type=item.get("requirementType"),
|
||||
sort_order=int(item.get("sortOrder") or index),
|
||||
)
|
||||
db.add(req)
|
||||
continue
|
||||
|
||||
req.title = item.get("title", req.title)
|
||||
req.description = item.get("description", req.description)
|
||||
req.priority = item.get("priority", req.priority)
|
||||
req.acceptance_criteria = item.get("acceptanceCriteria", req.acceptance_criteria)
|
||||
req.source_field = item.get("sourceField", req.source_field)
|
||||
req.section_number = item.get("sectionNumber", req.section_number)
|
||||
req.section_title = item.get("sectionTitle", req.section_title)
|
||||
req.requirement_type = item.get("requirementType", req.requirement_type)
|
||||
req.sort_order = int(item.get("sortOrder", index))
|
||||
|
||||
for uid, req in existing.items():
|
||||
if uid not in seen_ids:
|
||||
db.delete(req)
|
||||
|
||||
extraction.total_requirements = len(updates)
|
||||
extraction.statistics = {
|
||||
"total": len(updates),
|
||||
"by_type": _count_requirement_types(updates),
|
||||
}
|
||||
extraction.raw_output = {
|
||||
"document_name": extraction.document_name,
|
||||
"generated_at": extraction.generated_at.isoformat(),
|
||||
"requirements": updates,
|
||||
}
|
||||
|
||||
|
||||
def _count_requirement_types(items: List[Dict[str, Any]]) -> Dict[str, int]:
|
||||
stats: Dict[str, int] = {}
|
||||
for item in items:
|
||||
req_type = item.get("requirementType") or "functional"
|
||||
stats[req_type] = stats.get(req_type, 0) + 1
|
||||
return stats
|
||||
@@ -0,0 +1,3 @@
|
||||
from app.services.testing_pipeline.pipeline import run_testing_pipeline
|
||||
|
||||
__all__ = ["run_testing_pipeline"]
|
||||
20
rag-web-ui/backend/app/services/testing_pipeline/base.py
Normal file
20
rag-web-ui/backend/app/services/testing_pipeline/base.py
Normal file
@@ -0,0 +1,20 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolExecutionResult:
|
||||
context: Dict[str, Any]
|
||||
output_summary: str
|
||||
fallback_used: bool = False
|
||||
|
||||
|
||||
class TestingTool(ABC):
|
||||
name: str
|
||||
|
||||
@abstractmethod
|
||||
def execute(self, context: Dict[str, Any]) -> ToolExecutionResult:
|
||||
raise NotImplementedError
|
||||
99
rag-web-ui/backend/app/services/testing_pipeline/pipeline.py
Normal file
99
rag-web-ui/backend/app/services/testing_pipeline/pipeline.py
Normal file
@@ -0,0 +1,99 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from time import perf_counter
|
||||
from typing import Any, Dict, List, Optional
|
||||
from uuid import uuid4
|
||||
|
||||
from app.services.llm.llm_factory import LLMFactory
|
||||
from app.services.testing_pipeline.tools import build_default_tool_chain
|
||||
|
||||
|
||||
def _build_input_summary(context: Dict[str, Any]) -> str:
|
||||
req_text = str(context.get("user_requirement_text", "")).strip()
|
||||
req_type = str(context.get("requirement_type_input", "")).strip() or "auto"
|
||||
short_text = req_text if len(req_text) <= 60 else f"{req_text[:60]}..."
|
||||
return f"requirement_type_input={req_type}; requirement_text={short_text}"
|
||||
|
||||
|
||||
def _build_output_summary(context: Dict[str, Any]) -> str:
|
||||
req_type_result = context.get("requirement_type_result", {})
|
||||
req_type = req_type_result.get("requirement_type", "")
|
||||
test_items = context.get("test_items", {})
|
||||
test_cases = context.get("test_cases", {})
|
||||
|
||||
return (
|
||||
f"requirement_type={req_type}; "
|
||||
f"items={len(test_items.get('normal', [])) + len(test_items.get('abnormal', []))}; "
|
||||
f"cases={len(test_cases.get('normal', [])) + len(test_cases.get('abnormal', []))}"
|
||||
)
|
||||
|
||||
|
||||
def run_testing_pipeline(
|
||||
user_requirement_text: str,
|
||||
requirement_type_input: Optional[str] = None,
|
||||
debug: bool = False,
|
||||
knowledge_context: Optional[str] = None,
|
||||
use_model_generation: bool = False,
|
||||
max_items_per_group: int = 12,
|
||||
cases_per_item: int = 2,
|
||||
max_focus_points: int = 6,
|
||||
max_llm_calls: int = 10,
|
||||
) -> Dict[str, Any]:
|
||||
llm_model = None
|
||||
if use_model_generation:
|
||||
try:
|
||||
llm_model = LLMFactory.create(streaming=False)
|
||||
except Exception:
|
||||
llm_model = None
|
||||
|
||||
context: Dict[str, Any] = {
|
||||
"trace_id": str(uuid4()),
|
||||
"user_requirement_text": user_requirement_text,
|
||||
"requirement_type_input": requirement_type_input,
|
||||
"debug": bool(debug),
|
||||
"knowledge_context": (knowledge_context or "").strip(),
|
||||
"knowledge_used": bool((knowledge_context or "").strip()),
|
||||
"use_model_generation": bool(use_model_generation),
|
||||
"llm_model": llm_model,
|
||||
"max_items_per_group": max(4, min(int(max_items_per_group), 30)),
|
||||
"cases_per_item": max(1, min(int(cases_per_item), 5)),
|
||||
"max_focus_points": max(3, min(int(max_focus_points), 12)),
|
||||
"llm_call_budget": max(0, min(int(max_llm_calls), 100)),
|
||||
}
|
||||
|
||||
step_logs: List[Dict[str, Any]] = []
|
||||
|
||||
for tool in build_default_tool_chain():
|
||||
start = perf_counter()
|
||||
input_summary = _build_input_summary(context)
|
||||
|
||||
execution = tool.execute(context)
|
||||
context = execution.context
|
||||
|
||||
duration_ms = (perf_counter() - start) * 1000
|
||||
step_logs.append(
|
||||
{
|
||||
"step_name": tool.name,
|
||||
"input_summary": input_summary,
|
||||
"output_summary": execution.output_summary,
|
||||
"success": True,
|
||||
"fallback_used": execution.fallback_used,
|
||||
"duration_ms": round(duration_ms, 3),
|
||||
}
|
||||
)
|
||||
|
||||
req_result = context.get("requirement_type_result", {})
|
||||
|
||||
return {
|
||||
"trace_id": context.get("trace_id"),
|
||||
"requirement_type": req_result.get("requirement_type", "未知类型"),
|
||||
"reason": req_result.get("reason", ""),
|
||||
"candidates": req_result.get("candidates", []),
|
||||
"test_items": context.get("test_items", {"normal": [], "abnormal": []}),
|
||||
"test_cases": context.get("test_cases", {"normal": [], "abnormal": []}),
|
||||
"expected_results": context.get("expected_results", {"normal": [], "abnormal": []}),
|
||||
"formatted_output": context.get("formatted_output", ""),
|
||||
"pipeline_summary": _build_output_summary(context),
|
||||
"knowledge_used": bool(context.get("knowledge_used", False)),
|
||||
"step_logs": step_logs if debug else [],
|
||||
}
|
||||
203
rag-web-ui/backend/app/services/testing_pipeline/rules.py
Normal file
203
rag-web-ui/backend/app/services/testing_pipeline/rules.py
Normal file
@@ -0,0 +1,203 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Dict, List
|
||||
|
||||
|
||||
REQUIREMENT_TYPES: List[str] = [
|
||||
"功能测试",
|
||||
"性能测试",
|
||||
"外部接口测试",
|
||||
"人机交互界面测试",
|
||||
"强度测试",
|
||||
"余量测试",
|
||||
"可靠性测试",
|
||||
"安全性测试",
|
||||
"恢复性测试",
|
||||
"边界测试",
|
||||
"安装性测试",
|
||||
"互操作性测试",
|
||||
"敏感性测试",
|
||||
"测试充分性要求",
|
||||
]
|
||||
|
||||
|
||||
|
||||
TYPE_SIGNAL_RULES: Dict[str, str] = {
|
||||
"功能测试": "关注功能需求逐项验证、业务流程正确性、输入输出行为、状态转换与边界值处理。",
|
||||
"性能测试": "关注处理精度、响应时间、处理数据量、系统协调性、负载潜力与运行占用空间。",
|
||||
"外部接口测试": "关注外部输入输出接口的格式、内容、协议与正常/异常交互表现。",
|
||||
"人机交互界面测试": "关注界面一致性、界面风格、操作流程、误操作健壮性与错误提示能力。",
|
||||
"强度测试": "关注系统在极限、超负荷、饱和和降级条件下的稳定性与承受能力。",
|
||||
"余量测试": "关注存储余量、输入输出通道余量、功能处理时间余量等资源裕度。",
|
||||
"可靠性测试": "关注真实或仿真环境下的失效等级、运行剖面、输入覆盖和长期稳定运行能力。",
|
||||
"安全性测试": "关注危险状态响应、安全关键部件、异常输入防护、非法访问阻断和数据完整性保护。",
|
||||
"恢复性测试": "关注故障探测、备用切换、系统状态保护与从无错误状态继续执行能力。",
|
||||
"边界测试": "关注输入输出域边界、状态转换端点、功能界限、性能界限与容量界限。",
|
||||
"安装性测试": "关注不同配置下安装卸载流程和安装规程执行正确性。",
|
||||
"互操作性测试": "关注多个软件并行运行时的互操作能力与协同正确性。",
|
||||
"敏感性测试": "关注有效输入类中可能引发不稳定或不正常处理的数据组合。",
|
||||
"测试充分性要求": "关注需求覆盖率、配置项覆盖、语句覆盖、分支覆盖及未覆盖分析确认。",
|
||||
}
|
||||
|
||||
|
||||
DECOMPOSE_FORCE_RULES: List[str] = [
|
||||
"每个软件功能至少应被正常测试与被认可的异常场景覆盖；复杂功能需继续细分。",
|
||||
"每个测试项必须语义完整、可直接执行。",
|
||||
"覆盖必须包含：正常流程、边界条件（适用时）、异常条件。",
|
||||
"粒度需适中，避免过粗或过细。",
|
||||
"对未知类型必须执行通用分解，并保持正常/异常分组。",
|
||||
"对需求说明未显式给出但在用户手册或操作手册体现的功能，也应补充测试项覆盖。",
|
||||
]
|
||||
|
||||
|
||||
REQUIREMENT_RULES: Dict[str, Dict[str, List[str]]] = {
|
||||
"功能测试": {
|
||||
"keywords": ["功能", "业务流程", "输入输出", "状态转换", "边界值"],
|
||||
"normal": [
|
||||
"正常覆盖功能主路径、基本数据类型、合法边界值与状态转换。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖非法输入、不规则输入、非法边界值与最坏情况。",
|
||||
],
|
||||
},
|
||||
"性能测试": {
|
||||
"keywords": ["性能", "处理精度", "响应时间", "处理数据量", "负载", "占用空间"],
|
||||
"normal": [
|
||||
"正常覆盖处理精度、响应时间、处理数据量与模块协调性。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖超负荷、软硬件限制、负载潜力上限与资源占用异常。",
|
||||
],
|
||||
},
|
||||
"外部接口测试": {
|
||||
"keywords": ["外部接口", "输入接口", "输出接口", "格式", "内容", "协议", "异常交互"],
|
||||
"normal": [
|
||||
"正常覆盖全部外部接口格式与内容正确性。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖每个输入输出接口的错误格式、错误内容与异常交互。",
|
||||
],
|
||||
},
|
||||
"人机交互界面测试": {
|
||||
"keywords": ["界面", "风格", "交互", "误操作", "错误提示", "操作流程"],
|
||||
"normal": [
|
||||
"正常覆盖界面风格一致性与标准操作流程。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖误操作、快速操作、非法输入、错误命令与错误流程提示。",
|
||||
],
|
||||
},
|
||||
"强度测试": {
|
||||
"keywords": ["强度", "极限", "超负荷", "饱和", "降级", "健壮性"],
|
||||
"normal": [
|
||||
"正常覆盖设计极限下系统功能和性能表现。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖超出极限时的降级行为、健壮性与饱和表现。",
|
||||
],
|
||||
},
|
||||
"余量测试": {
|
||||
"keywords": ["余量", "存储余量", "通道余量", "处理时间余量", "资源裕度"],
|
||||
"normal": [
|
||||
"正常覆盖存储、通道、处理时间余量是否满足要求。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖余量不足或耗尽时系统告警与受控行为。",
|
||||
],
|
||||
},
|
||||
"可靠性测试": {
|
||||
"keywords": ["可靠性", "运行剖面", "失效等级", "输入覆盖", "长期稳定"],
|
||||
"normal": [
|
||||
"正常覆盖典型环境、运行剖面与输入变量组合。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖失效等级场景、边界环境变化、不合法输入域及失效记录。",
|
||||
],
|
||||
},
|
||||
"安全性测试": {
|
||||
"keywords": ["安全", "危险状态", "安全关键部件", "非法进入", "完整性", "防护"],
|
||||
"normal": [
|
||||
"正常覆盖安全关键部件、安全结构与合法操作路径。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖危险状态、故障模式、边界接合部、非法进入与数据完整性保护。",
|
||||
],
|
||||
},
|
||||
"恢复性测试": {
|
||||
"keywords": ["恢复", "故障探测", "备用切换", "状态保护", "继续执行", "reset"],
|
||||
"normal": [
|
||||
"正常覆盖故障探测、备用切换、恢复后继续执行。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖故障中作业保护、状态保护与恢复失败路径。",
|
||||
],
|
||||
},
|
||||
"边界测试": {
|
||||
"keywords": ["边界", "端点", "输入输出域", "状态转换", "性能界限", "容量界限"],
|
||||
"normal": [
|
||||
"正常覆盖输入输出域边界、状态转换端点与功能界限。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖性能界限、容量界限和越界端点。",
|
||||
],
|
||||
},
|
||||
"安装性测试": {
|
||||
"keywords": ["安装", "卸载", "配置", "安装规程", "部署", "中断"],
|
||||
"normal": [
|
||||
"正常覆盖标准及不同配置下安装卸载流程。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖安装规程错误、依赖异常与中断后的处理。",
|
||||
],
|
||||
},
|
||||
"互操作性测试": {
|
||||
"keywords": ["互操作", "并行运行", "协同", "兼容", "冲突", "互操作失败"],
|
||||
"normal": [
|
||||
"正常覆盖两个或多个软件同时运行与互操作过程。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖互操作失败、并行冲突与协同异常。",
|
||||
],
|
||||
},
|
||||
"敏感性测试": {
|
||||
"keywords": ["敏感性", "输入类", "数据组合", "不稳定", "不正常处理"],
|
||||
"normal": [
|
||||
"正常覆盖有效输入类中典型数据组合。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖引发不稳定或不正常处理的特殊数据组合。",
|
||||
],
|
||||
},
|
||||
"测试充分性要求": {
|
||||
"keywords": ["测试充分性", "需求覆盖率", "配置项覆盖", "语句覆盖", "分支覆盖", "未覆盖分析"],
|
||||
"normal": [
|
||||
"正常覆盖需求覆盖率、配置项覆盖与代码覆盖达标。",
|
||||
],
|
||||
"abnormal": [
|
||||
"异常覆盖未覆盖部分逐项分析、确认与报告输出。",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
GENERIC_DECOMPOSITION_RULES: Dict[str, List[str]] = {
|
||||
"normal": [
|
||||
"主流程正确性。",
|
||||
"合法边界值。",
|
||||
"标准输入输出。",
|
||||
],
|
||||
"abnormal": [
|
||||
"非法输入。",
|
||||
"越界输入。",
|
||||
"资源异常或状态冲突。",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
EXPECTED_RESULT_PLACEHOLDER_MAP: Dict[str, str] = {
|
||||
"{{return_value}}": "接口或函数返回值验证。",
|
||||
"{{state_change}}": "系统状态变化验证。",
|
||||
"{{error_message}}": "异常场景错误信息验证。",
|
||||
"{{data_persistence}}": "数据库或存储落库结果验证。",
|
||||
"{{ui_display}}": "界面显示反馈验证。",
|
||||
}
|
||||
867
rag-web-ui/backend/app/services/testing_pipeline/tools.py
Normal file
867
rag-web-ui/backend/app/services/testing_pipeline/tools.py
Normal file
@@ -0,0 +1,867 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from app.services.testing_pipeline.base import TestingTool, ToolExecutionResult
|
||||
from app.services.testing_pipeline.rules import (
|
||||
DECOMPOSE_FORCE_RULES,
|
||||
EXPECTED_RESULT_PLACEHOLDER_MAP,
|
||||
GENERIC_DECOMPOSITION_RULES,
|
||||
REQUIREMENT_RULES,
|
||||
REQUIREMENT_TYPES,
|
||||
TYPE_SIGNAL_RULES,
|
||||
)
|
||||
|
||||
|
||||
def _clean_text(value: str) -> str:
|
||||
return " ".join((value or "").replace("\n", " ").split())
|
||||
|
||||
|
||||
def _truncate_text(value: str, max_len: int = 2000) -> str:
|
||||
text = _clean_text(value)
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return f"{text[:max_len]}..."
|
||||
|
||||
|
||||
def _safe_int(value: Any, default: int, low: int, high: int) -> int:
|
||||
try:
|
||||
parsed = int(value)
|
||||
except Exception:
|
||||
parsed = default
|
||||
return max(low, min(parsed, high))
|
||||
|
||||
|
||||
def _strip_instruction_prefix(value: str) -> str:
|
||||
text = _clean_text(value)
|
||||
if not text:
|
||||
return text
|
||||
|
||||
lowered = text.lower()
|
||||
if lowered.startswith("/testing"):
|
||||
text = _clean_text(text[len("/testing") :])
|
||||
|
||||
prefixes = [
|
||||
"为以下需求生成测试用例",
|
||||
"根据以下需求生成测试用例",
|
||||
"请根据以下需求生成测试用例",
|
||||
"请根据需求生成测试用例",
|
||||
"请生成测试用例",
|
||||
"生成测试用例",
|
||||
]
|
||||
for prefix in prefixes:
|
||||
if text.startswith(prefix):
|
||||
for sep in ("：", ":"):
|
||||
idx = text.find(sep)
|
||||
if idx != -1:
|
||||
text = _clean_text(text[idx + 1 :])
|
||||
break
|
||||
else:
|
||||
text = _clean_text(text[len(prefix) :])
|
||||
break
|
||||
|
||||
pattern = re.compile(r"^(请)?(根据|按|基于).{0,40}(需求|场景).{0,30}(生成|输出).{0,20}(测试项|测试用例)[：:]")
|
||||
matched = pattern.match(text)
|
||||
if matched:
|
||||
text = _clean_text(text[matched.end() :])
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def _extract_focus_points(value: str, max_points: int = 6) -> List[str]:
|
||||
text = _strip_instruction_prefix(value)
|
||||
if not text:
|
||||
return []
|
||||
|
||||
parts = [_clean_text(part) for part in re.split(r"[，,。；;]", text)]
|
||||
parts = [part for part in parts if part]
|
||||
|
||||
ignored_tokens = ["生成测试用例", "测试项分解", "测试用例生成", "以下需求"]
|
||||
filtered = [
|
||||
part
|
||||
for part in parts
|
||||
if len(part) >= 4 and not any(token in part for token in ignored_tokens)
|
||||
]
|
||||
if not filtered:
|
||||
filtered = parts
|
||||
|
||||
priority_keywords = [
|
||||
"启停",
|
||||
"开启",
|
||||
"关闭",
|
||||
"远程控制",
|
||||
"保护",
|
||||
"联动",
|
||||
"状态",
|
||||
"故障",
|
||||
"恢复",
|
||||
"切换",
|
||||
"告警",
|
||||
"模式",
|
||||
"边界",
|
||||
"时序",
|
||||
]
|
||||
priority = [part for part in filtered if any(keyword in part for keyword in priority_keywords)]
|
||||
candidates = priority if priority else filtered
|
||||
|
||||
unique: List[str] = []
|
||||
for part in candidates:
|
||||
if part not in unique:
|
||||
unique.append(part)
|
||||
|
||||
return unique[:max_points]
|
||||
|
||||
|
||||
def _build_type_scores(text: str) -> Dict[str, int]:
|
||||
scores: Dict[str, int] = {}
|
||||
lowered = text.lower()
|
||||
|
||||
for req_type, rule in REQUIREMENT_RULES.items():
|
||||
score = 0
|
||||
if req_type in text:
|
||||
score += 5
|
||||
for keyword in rule.get("keywords", []):
|
||||
if keyword.lower() in lowered:
|
||||
score += 2
|
||||
scores[req_type] = score
|
||||
|
||||
return scores
|
||||
|
||||
|
||||
def _top_candidates(scores: Dict[str, int], top_n: int = 3) -> List[str]:
|
||||
sorted_pairs = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
|
||||
non_zero = [name for name, score in sorted_pairs if score > 0]
|
||||
if non_zero:
|
||||
return non_zero[:top_n]
|
||||
return ["功能测试", "边界测试", "性能测试"][:top_n]
|
||||
|
||||
|
||||
def _message_to_text(value: Any) -> str:
|
||||
content = getattr(value, "content", value)
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
chunks: List[str] = []
|
||||
for item in content:
|
||||
if isinstance(item, str):
|
||||
chunks.append(item)
|
||||
elif isinstance(item, dict):
|
||||
text = item.get("text")
|
||||
if isinstance(text, str):
|
||||
chunks.append(text)
|
||||
else:
|
||||
chunks.append(str(item))
|
||||
return "".join(chunks)
|
||||
return str(content)
|
||||
|
||||
|
||||
def _extract_json_object(value: str) -> Optional[Dict[str, Any]]:
|
||||
text = (value or "").strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
if text.startswith("```"):
|
||||
text = re.sub(r"^```(?:json)?", "", text, flags=re.IGNORECASE).strip()
|
||||
if text.endswith("```"):
|
||||
text = text[:-3].strip()
|
||||
|
||||
try:
|
||||
data = json.loads(text)
|
||||
if isinstance(data, dict):
|
||||
return data
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
start = text.find("{")
|
||||
if start == -1:
|
||||
return None
|
||||
|
||||
depth = 0
|
||||
for idx in range(start, len(text)):
|
||||
ch = text[idx]
|
||||
if ch == "{":
|
||||
depth += 1
|
||||
elif ch == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
fragment = text[start : idx + 1]
|
||||
try:
|
||||
data = json.loads(fragment)
|
||||
if isinstance(data, dict):
|
||||
return data
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _invoke_llm_json(context: Dict[str, Any], prompt: str) -> Optional[Dict[str, Any]]:
|
||||
model = context.get("llm_model")
|
||||
if model is None or not context.get("use_model_generation"):
|
||||
return None
|
||||
|
||||
budget = context.get("llm_call_budget")
|
||||
if isinstance(budget, int):
|
||||
if budget <= 0:
|
||||
return None
|
||||
context["llm_call_budget"] = budget - 1
|
||||
|
||||
try:
|
||||
response = model.invoke(prompt)
|
||||
text = _message_to_text(response)
|
||||
return _extract_json_object(text)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _invoke_llm_text(context: Dict[str, Any], prompt: str) -> str:
|
||||
model = context.get("llm_model")
|
||||
if model is None or not context.get("use_model_generation"):
|
||||
return ""
|
||||
|
||||
budget = context.get("llm_call_budget")
|
||||
if isinstance(budget, int):
|
||||
if budget <= 0:
|
||||
return ""
|
||||
context["llm_call_budget"] = budget - 1
|
||||
|
||||
try:
|
||||
response = model.invoke(prompt)
|
||||
return _clean_text(_message_to_text(response))
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _normalize_item_entry(item: Any) -> Optional[Dict[str, Any]]:
|
||||
if isinstance(item, str):
|
||||
content = _clean_text(item)
|
||||
if not content:
|
||||
return None
|
||||
return {"content": content, "coverage_tags": []}
|
||||
|
||||
if isinstance(item, dict):
|
||||
content = _clean_text(str(item.get("content", "")))
|
||||
if not content:
|
||||
return None
|
||||
tags = item.get("coverage_tags") or item.get("covered_points") or []
|
||||
if not isinstance(tags, list):
|
||||
tags = [str(tags)]
|
||||
tags = [_clean_text(str(tag)) for tag in tags if _clean_text(str(tag))]
|
||||
return {"content": content, "coverage_tags": tags}
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _dedupe_items(items: List[Dict[str, Any]], max_items: int) -> List[Dict[str, Any]]:
|
||||
merged: Dict[str, Dict[str, Any]] = {}
|
||||
for item in items:
|
||||
content = _clean_text(item.get("content", ""))
|
||||
if not content:
|
||||
continue
|
||||
existing = merged.get(content)
|
||||
if existing is None:
|
||||
merged[content] = {
|
||||
"content": content,
|
||||
"coverage_tags": list(item.get("coverage_tags") or []),
|
||||
}
|
||||
else:
|
||||
existing_tags = set(existing.get("coverage_tags") or [])
|
||||
for tag in item.get("coverage_tags") or []:
|
||||
if tag and tag not in existing_tags:
|
||||
existing_tags.add(tag)
|
||||
existing["coverage_tags"] = list(existing_tags)
|
||||
|
||||
deduped = list(merged.values())
|
||||
return deduped[:max_items]
|
||||
|
||||
|
||||
def _pick_expected_result_placeholder(content: str, abnormal: bool) -> str:
|
||||
text = content or ""
|
||||
|
||||
if abnormal or any(token in text for token in ["非法", "异常", "错误", "拒绝", "越界", "失败"]):
|
||||
return "{{error_message}}"
|
||||
if any(token in text for token in ["状态", "切换", "转换", "恢复"]):
|
||||
return "{{state_change}}"
|
||||
if any(token in text for token in ["数据库", "存储", "落库", "持久化"]):
|
||||
return "{{data_persistence}}"
|
||||
if any(token in text for token in ["界面", "UI", "页面", "按钮", "提示"]):
|
||||
return "{{ui_display}}"
|
||||
return "{{return_value}}"
|
||||
|
||||
|
||||
class IdentifyRequirementTypeTool(TestingTool):
|
||||
name = "identify-requirement-type"
|
||||
|
||||
def execute(self, context: Dict[str, Any]) -> ToolExecutionResult:
|
||||
raw_text = _clean_text(context.get("user_requirement_text", ""))
|
||||
text = _strip_instruction_prefix(raw_text)
|
||||
if not text:
|
||||
text = raw_text
|
||||
|
||||
max_focus_points = _safe_int(context.get("max_focus_points"), 6, 3, 12)
|
||||
provided_type = _clean_text(context.get("requirement_type_input", ""))
|
||||
focus_points = _extract_focus_points(text, max_points=max_focus_points)
|
||||
fallback_used = False
|
||||
|
||||
if provided_type in REQUIREMENT_TYPES:
|
||||
result = {
|
||||
"requirement_type": provided_type,
|
||||
"reason": "用户已显式指定需求类型，系统按指定类型执行。",
|
||||
"candidates": [],
|
||||
"scores": {},
|
||||
"secondary_types": [],
|
||||
}
|
||||
else:
|
||||
scores = _build_type_scores(text)
|
||||
sorted_pairs = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
|
||||
best_type, best_score = sorted_pairs[0]
|
||||
secondary = [name for name, score in sorted_pairs[1:4] if score > 0]
|
||||
|
||||
if best_score <= 0:
|
||||
fallback_used = True
|
||||
candidates = _top_candidates(scores)
|
||||
result = {
|
||||
"requirement_type": "未知类型",
|
||||
"reason": "未命中明确分类规则，已回退到未知类型并提供最接近候选。",
|
||||
"candidates": candidates,
|
||||
"scores": scores,
|
||||
"secondary_types": [],
|
||||
}
|
||||
else:
|
||||
signal = TYPE_SIGNAL_RULES.get(best_type, "")
|
||||
result = {
|
||||
"requirement_type": best_type,
|
||||
"reason": f"命中{best_type}识别信号。{signal}",
|
||||
"candidates": [],
|
||||
"scores": scores,
|
||||
"secondary_types": secondary,
|
||||
}
|
||||
|
||||
context["requirement_type_result"] = result
|
||||
context["normalized_requirement_text"] = text
|
||||
context["requirement_focus_points"] = focus_points
|
||||
context["knowledge_used"] = bool(context.get("knowledge_context"))
|
||||
|
||||
return ToolExecutionResult(
|
||||
context=context,
|
||||
output_summary=(
|
||||
f"type={result['requirement_type']}; candidates={len(result['candidates'])}; "
|
||||
f"secondary_types={len(result.get('secondary_types', []))}; focus_points={len(focus_points)}"
|
||||
),
|
||||
fallback_used=fallback_used,
|
||||
)
|
||||
|
||||
|
||||
class DecomposeTestItemsTool(TestingTool):
|
||||
name = "decompose-test-items"
|
||||
|
||||
@staticmethod
|
||||
def _seed_items(
|
||||
req_type: str,
|
||||
req_text: str,
|
||||
focus_points: List[str],
|
||||
max_items: int,
|
||||
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
|
||||
if req_type in REQUIREMENT_RULES:
|
||||
source_rules = REQUIREMENT_RULES[req_type]
|
||||
normal_templates = list(source_rules.get("normal", []))
|
||||
abnormal_templates = list(source_rules.get("abnormal", []))
|
||||
else:
|
||||
normal_templates = list(GENERIC_DECOMPOSITION_RULES["normal"])
|
||||
abnormal_templates = list(GENERIC_DECOMPOSITION_RULES["abnormal"])
|
||||
|
||||
normal: List[Dict[str, Any]] = []
|
||||
abnormal: List[Dict[str, Any]] = []
|
||||
|
||||
for template in normal_templates:
|
||||
normal.append({"content": template, "coverage_tags": [req_type]})
|
||||
for template in abnormal_templates:
|
||||
abnormal.append({"content": template, "coverage_tags": [req_type]})
|
||||
|
||||
for point in focus_points:
|
||||
normal.extend(
|
||||
[
|
||||
{
|
||||
"content": f"验证{point}在标准作业流程下稳定执行且结果符合业务约束。",
|
||||
"coverage_tags": [point, "正常流程"],
|
||||
},
|
||||
{
|
||||
"content": f"验证{point}与相关联动控制、状态同步和回执反馈的一致性。",
|
||||
"coverage_tags": [point, "联动一致性"],
|
||||
},
|
||||
]
|
||||
)
|
||||
abnormal.extend(
|
||||
[
|
||||
{
|
||||
"content": f"验证{point}在非法输入、错误指令或权限异常时的保护与拒绝机制。",
|
||||
"coverage_tags": [point, "异常输入"],
|
||||
},
|
||||
{
|
||||
"content": f"验证{point}在边界条件、时序冲突或设备故障下的告警和恢复行为。",
|
||||
"coverage_tags": [point, "边界异常"],
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
if any(token in req_text for token in ["手册", "操作手册", "用户手册", "作业指导"]):
|
||||
normal.append(
|
||||
{
|
||||
"content": "验证需求说明未显式给出但在用户手册或操作手册体现的功能流程。",
|
||||
"coverage_tags": ["手册功能"],
|
||||
}
|
||||
)
|
||||
|
||||
return _dedupe_items(normal, max_items), _dedupe_items(abnormal, max_items)
|
||||
|
||||
@staticmethod
|
||||
def _generate_by_llm(context: Dict[str, Any]) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
|
||||
req_result = context.get("requirement_type_result", {})
|
||||
req_type = req_result.get("requirement_type", "未知类型")
|
||||
req_text = context.get("normalized_requirement_text", "")
|
||||
focus_points = context.get("requirement_focus_points", [])
|
||||
max_items = _safe_int(context.get("max_items_per_group"), 12, 4, 30)
|
||||
knowledge_context = _truncate_text(context.get("knowledge_context", ""), max_len=2500)
|
||||
|
||||
prompt = f"""
|
||||
你是资深测试分析师。请根据需求、分解规则和知识库片段，生成尽可能覆盖要点的测试项。
|
||||
|
||||
需求文本：{req_text}
|
||||
需求类型：{req_type}
|
||||
需求要点：{focus_points}
|
||||
知识库片段：{knowledge_context or '无'}
|
||||
|
||||
分解约束：
|
||||
1. 正常测试与异常测试必须分组输出。
|
||||
2. 每条测试项必须可执行、可验证，避免模板化空话。
|
||||
3. 尽可能覆盖全部需求要点；每组建议输出6-{max_items}条。
|
||||
4. 优先生成与需求对象/控制逻辑/异常处理/边界条件强相关的测试项。
|
||||
|
||||
请仅输出 JSON 对象，结构如下：
|
||||
{{
|
||||
"normal_test_items": [
|
||||
{{"content": "...", "coverage_tags": ["..."]}}
|
||||
],
|
||||
"abnormal_test_items": [
|
||||
{{"content": "...", "coverage_tags": ["..."]}}
|
||||
]
|
||||
}}
|
||||
""".strip()
|
||||
|
||||
data = _invoke_llm_json(context, prompt)
|
||||
if not data:
|
||||
return [], []
|
||||
|
||||
normal_raw = data.get("normal_test_items", [])
|
||||
abnormal_raw = data.get("abnormal_test_items", [])
|
||||
|
||||
normal: List[Dict[str, Any]] = []
|
||||
abnormal: List[Dict[str, Any]] = []
|
||||
|
||||
for item in normal_raw if isinstance(normal_raw, list) else []:
|
||||
normalized = _normalize_item_entry(item)
|
||||
if normalized:
|
||||
normal.append(normalized)
|
||||
|
||||
for item in abnormal_raw if isinstance(abnormal_raw, list) else []:
|
||||
normalized = _normalize_item_entry(item)
|
||||
if normalized:
|
||||
abnormal.append(normalized)
|
||||
|
||||
return _dedupe_items(normal, max_items), _dedupe_items(abnormal, max_items)
|
||||
|
||||
def execute(self, context: Dict[str, Any]) -> ToolExecutionResult:
|
||||
req_result = context.get("requirement_type_result", {})
|
||||
req_type = req_result.get("requirement_type", "未知类型")
|
||||
req_text = context.get("normalized_requirement_text") or _strip_instruction_prefix(
|
||||
context.get("user_requirement_text", "")
|
||||
)
|
||||
focus_points = context.get("requirement_focus_points", [])
|
||||
max_items = _safe_int(context.get("max_items_per_group"), 12, 4, 30)
|
||||
|
||||
seeded_normal, seeded_abnormal = self._seed_items(req_type, req_text, focus_points, max_items)
|
||||
llm_normal, llm_abnormal = self._generate_by_llm(context)
|
||||
|
||||
merged_normal = _dedupe_items(llm_normal + seeded_normal, max_items)
|
||||
merged_abnormal = _dedupe_items(llm_abnormal + seeded_abnormal, max_items)
|
||||
|
||||
fallback_used = not bool(llm_normal or llm_abnormal)
|
||||
|
||||
normal_items: List[Dict[str, Any]] = []
|
||||
abnormal_items: List[Dict[str, Any]] = []
|
||||
|
||||
for idx, item in enumerate(merged_normal, start=1):
|
||||
normal_items.append(
|
||||
{
|
||||
"id": f"N{idx}",
|
||||
"content": item["content"],
|
||||
"coverage_tags": item.get("coverage_tags", []),
|
||||
}
|
||||
)
|
||||
|
||||
for idx, item in enumerate(merged_abnormal, start=1):
|
||||
abnormal_items.append(
|
||||
{
|
||||
"id": f"E{idx}",
|
||||
"content": item["content"],
|
||||
"coverage_tags": item.get("coverage_tags", []),
|
||||
}
|
||||
)
|
||||
|
||||
context["test_items"] = {
|
||||
"normal": normal_items,
|
||||
"abnormal": abnormal_items,
|
||||
}
|
||||
context["decompose_force_rules"] = DECOMPOSE_FORCE_RULES
|
||||
|
||||
return ToolExecutionResult(
|
||||
context=context,
|
||||
output_summary=(
|
||||
f"normal_items={len(normal_items)}; abnormal_items={len(abnormal_items)}; "
|
||||
f"llm_items={len(llm_normal) + len(llm_abnormal)}"
|
||||
),
|
||||
fallback_used=fallback_used,
|
||||
)
|
||||
|
||||
|
||||
class GenerateTestCasesTool(TestingTool):
|
||||
name = "generate-test-cases"
|
||||
|
||||
@staticmethod
|
||||
def _build_fallback_steps(item_content: str, abnormal: bool, variant: str) -> List[str]:
|
||||
if abnormal:
|
||||
return [
|
||||
"确认测试前置环境、设备状态与日志采集开关已准备就绪。",
|
||||
f"准备异常场景“{variant}”所需的输入数据、操作账号和触发条件。",
|
||||
f"在目标对象执行异常触发操作，重点验证：{item_content}",
|
||||
"持续观察系统返回码、错误文案、告警信息与日志链路完整性。",
|
||||
"检查保护机制是否生效，包括拒绝策略、回滚行为和状态一致性。",
|
||||
"记录证据并复位环境，确认异常处理后系统可恢复到稳定状态。",
|
||||
]
|
||||
|
||||
return [
|
||||
"确认测试环境、设备连接状态和前置业务数据均已初始化。",
|
||||
f"准备“{variant}”所需输入参数、操作路径和判定阈值。",
|
||||
f"在目标对象执行业务控制流程，重点验证：{item_content}",
|
||||
"校验关键返回值、状态变化、控制回执及界面或接口反馈结果。",
|
||||
"检查联动模块、日志记录和数据落库是否满足一致性要求。",
|
||||
"沉淀测试证据并恢复环境，确保后续用例可重复执行。",
|
||||
]
|
||||
|
||||
def _generate_cases_by_llm(
|
||||
self,
|
||||
context: Dict[str, Any],
|
||||
item: Dict[str, Any],
|
||||
abnormal: bool,
|
||||
cases_per_item: int,
|
||||
) -> List[Dict[str, Any]]:
|
||||
req_text = context.get("normalized_requirement_text", "")
|
||||
knowledge_context = _truncate_text(context.get("knowledge_context", ""), max_len=1800)
|
||||
|
||||
prompt = f"""
|
||||
你是资深测试工程师。请围绕给定测试项生成详细测试用例。
|
||||
|
||||
需求：{req_text}
|
||||
测试项：{item.get('content', '')}
|
||||
测试类型：{'异常测试' if abnormal else '正常测试'}
|
||||
知识库片段：{knowledge_context or '无'}
|
||||
|
||||
要求：
|
||||
1. 生成 {cases_per_item}-{max(cases_per_item + 1, cases_per_item)} 条测试用例。
|
||||
2. 每条用例包含 test_content 与 operation_steps。
|
||||
3. operation_steps 必须详细，至少5步，包含前置、执行、观察、校验与证据留存。
|
||||
4. 内容必须围绕当前测试项，不要输出空洞模板。
|
||||
|
||||
仅输出 JSON：
|
||||
{{
|
||||
"test_cases": [
|
||||
{{
|
||||
"title": "...",
|
||||
"test_content": "...",
|
||||
"operation_steps": ["...", "..."]
|
||||
}}
|
||||
]
|
||||
}}
|
||||
""".strip()
|
||||
|
||||
data = _invoke_llm_json(context, prompt)
|
||||
if not data:
|
||||
return []
|
||||
|
||||
raw_cases = data.get("test_cases", [])
|
||||
if not isinstance(raw_cases, list):
|
||||
return []
|
||||
|
||||
normalized_cases: List[Dict[str, Any]] = []
|
||||
for case in raw_cases:
|
||||
if not isinstance(case, dict):
|
||||
continue
|
||||
test_content = _clean_text(str(case.get("test_content", "")))
|
||||
if not test_content:
|
||||
continue
|
||||
steps = case.get("operation_steps", [])
|
||||
if not isinstance(steps, list):
|
||||
continue
|
||||
cleaned_steps = [_clean_text(str(step)) for step in steps if _clean_text(str(step))]
|
||||
if len(cleaned_steps) < 5:
|
||||
continue
|
||||
normalized_cases.append(
|
||||
{
|
||||
"title": _clean_text(str(case.get("title", ""))),
|
||||
"test_content": test_content,
|
||||
"operation_steps": cleaned_steps,
|
||||
}
|
||||
)
|
||||
|
||||
return normalized_cases[: max(1, cases_per_item)]
|
||||
|
||||
def execute(self, context: Dict[str, Any]) -> ToolExecutionResult:
|
||||
test_items = context.get("test_items", {})
|
||||
cases_per_item = _safe_int(context.get("cases_per_item"), 2, 1, 5)
|
||||
|
||||
normal_cases: List[Dict[str, Any]] = []
|
||||
abnormal_cases: List[Dict[str, Any]] = []
|
||||
llm_case_count = 0
|
||||
|
||||
for item in test_items.get("normal", []):
|
||||
generated = self._generate_cases_by_llm(context, item, abnormal=False, cases_per_item=cases_per_item)
|
||||
if not generated:
|
||||
generated = [
|
||||
{
|
||||
"title": "标准流程验证",
|
||||
"test_content": f"验证{item['content']}",
|
||||
"operation_steps": self._build_fallback_steps(item["content"], False, "标准流程"),
|
||||
},
|
||||
{
|
||||
"title": "边界与联动验证",
|
||||
"test_content": f"验证{item['content']}在边界条件和联动场景下的稳定性",
|
||||
"operation_steps": self._build_fallback_steps(item["content"], False, "边界与联动"),
|
||||
},
|
||||
][:cases_per_item]
|
||||
else:
|
||||
llm_case_count += len(generated)
|
||||
|
||||
for idx, case in enumerate(generated, start=1):
|
||||
merged_content = _clean_text(case.get("test_content", item["content"]))
|
||||
placeholder = _pick_expected_result_placeholder(merged_content, abnormal=False)
|
||||
normal_cases.append(
|
||||
{
|
||||
"id": f"{item['id']}-C{idx}",
|
||||
"item_id": item["id"],
|
||||
"title": _clean_text(case.get("title", "")),
|
||||
"operation_steps": case.get("operation_steps", []),
|
||||
"test_content": merged_content,
|
||||
"expected_result_placeholder": placeholder,
|
||||
}
|
||||
)
|
||||
|
||||
for item in test_items.get("abnormal", []):
|
||||
generated = self._generate_cases_by_llm(context, item, abnormal=True, cases_per_item=cases_per_item)
|
||||
if not generated:
|
||||
generated = [
|
||||
{
|
||||
"title": "非法输入与权限异常验证",
|
||||
"test_content": f"验证{item['content']}在非法输入与权限异常下的处理表现",
|
||||
"operation_steps": self._build_fallback_steps(item["content"], True, "非法输入与权限异常"),
|
||||
},
|
||||
{
|
||||
"title": "故障与时序冲突验证",
|
||||
"test_content": f"验证{item['content']}在故障和时序冲突场景下的保护行为",
|
||||
"operation_steps": self._build_fallback_steps(item["content"], True, "故障与时序冲突"),
|
||||
},
|
||||
][:cases_per_item]
|
||||
else:
|
||||
llm_case_count += len(generated)
|
||||
|
||||
for idx, case in enumerate(generated, start=1):
|
||||
merged_content = _clean_text(case.get("test_content", item["content"]))
|
||||
placeholder = _pick_expected_result_placeholder(merged_content, abnormal=True)
|
||||
abnormal_cases.append(
|
||||
{
|
||||
"id": f"{item['id']}-C{idx}",
|
||||
"item_id": item["id"],
|
||||
"title": _clean_text(case.get("title", "")),
|
||||
"operation_steps": case.get("operation_steps", []),
|
||||
"test_content": merged_content,
|
||||
"expected_result_placeholder": placeholder,
|
||||
}
|
||||
)
|
||||
|
||||
context["test_cases"] = {
|
||||
"normal": normal_cases,
|
||||
"abnormal": abnormal_cases,
|
||||
}
|
||||
|
||||
return ToolExecutionResult(
|
||||
context=context,
|
||||
output_summary=(
|
||||
f"normal_cases={len(normal_cases)}; abnormal_cases={len(abnormal_cases)}; llm_cases={llm_case_count}"
|
||||
),
|
||||
fallback_used=llm_case_count == 0,
|
||||
)
|
||||
|
||||
|
||||
class BuildExpectedResultsTool(TestingTool):
|
||||
name = "build_expected_results"
|
||||
|
||||
def _expected_for_case(self, context: Dict[str, Any], case: Dict[str, Any], abnormal: bool) -> str:
|
||||
placeholder = case.get("expected_result_placeholder", "{{return_value}}")
|
||||
if placeholder not in EXPECTED_RESULT_PLACEHOLDER_MAP:
|
||||
placeholder = "{{return_value}}"
|
||||
|
||||
req_text = context.get("normalized_requirement_text", "")
|
||||
knowledge_context = _truncate_text(context.get("knowledge_context", ""), max_len=1200)
|
||||
prompt = f"""
|
||||
请基于以下信息生成一条可验证、可度量的测试预期结果，避免模板化空话。
|
||||
|
||||
需求：{req_text}
|
||||
测试内容：{case.get('test_content', '')}
|
||||
测试类型：{'异常测试' if abnormal else '正常测试'}
|
||||
占位符语义：{placeholder} -> {EXPECTED_RESULT_PLACEHOLDER_MAP.get(placeholder, '')}
|
||||
知识库片段：{knowledge_context or '无'}
|
||||
|
||||
输出要求：
|
||||
1. 仅输出一句中文预期结果。
|
||||
2. 结果必须可判定成功/失败。
|
||||
3. 包含关键观测项（返回值、状态、告警、日志、数据一致性中的相关项）。
|
||||
""".strip()
|
||||
|
||||
llm_text = _invoke_llm_text(context, prompt)
|
||||
if llm_text:
|
||||
return _truncate_text(llm_text, max_len=220)
|
||||
|
||||
test_content = _clean_text(case.get("test_content", ""))
|
||||
if placeholder == "{{error_message}}":
|
||||
return f"触发{test_content}后，系统应返回明确错误码与错误文案，拒绝非法请求且核心状态保持一致。"
|
||||
if placeholder == "{{state_change}}":
|
||||
return f"执行{test_content}后，系统状态转换应符合需求定义，状态变化可被日志与回执共同验证。"
|
||||
if placeholder == "{{data_persistence}}":
|
||||
return f"执行{test_content}后，数据库或存储层应产生符合约束的持久化结果且无脏数据。"
|
||||
if placeholder == "{{ui_display}}":
|
||||
return f"执行{test_content}后，界面应展示与控制结果一致的反馈信息且提示可被用户执行。"
|
||||
|
||||
if abnormal:
|
||||
return f"执行异常场景“{test_content}”后，系统应触发保护策略并输出可追溯日志，业务状态保持可恢复。"
|
||||
|
||||
return f"执行“{test_content}”后，返回值与状态变化应满足需求约束，关键结果可通过日志或回执验证。"
|
||||
|
||||
def execute(self, context: Dict[str, Any]) -> ToolExecutionResult:
|
||||
test_cases = context.get("test_cases", {})
|
||||
|
||||
normal_expected: List[Dict[str, str]] = []
|
||||
abnormal_expected: List[Dict[str, str]] = []
|
||||
|
||||
for case in test_cases.get("normal", []):
|
||||
normal_expected.append(
|
||||
{
|
||||
"id": case["id"],
|
||||
"case_id": case["id"],
|
||||
"result": self._expected_for_case(context, case, abnormal=False),
|
||||
}
|
||||
)
|
||||
|
||||
for case in test_cases.get("abnormal", []):
|
||||
abnormal_expected.append(
|
||||
{
|
||||
"id": case["id"],
|
||||
"case_id": case["id"],
|
||||
"result": self._expected_for_case(context, case, abnormal=True),
|
||||
}
|
||||
)
|
||||
|
||||
context["expected_results"] = {
|
||||
"normal": normal_expected,
|
||||
"abnormal": abnormal_expected,
|
||||
}
|
||||
|
||||
return ToolExecutionResult(
|
||||
context=context,
|
||||
output_summary=(
|
||||
f"normal_expected={len(normal_expected)}; abnormal_expected={len(abnormal_expected)}"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class FormatOutputTool(TestingTool):
|
||||
name = "format_output"
|
||||
|
||||
@staticmethod
|
||||
def _format_case_block(case: Dict[str, Any], index: int) -> List[str]:
|
||||
item_id = case.get("item_id", case.get("id", ""))
|
||||
title = _clean_text(case.get("title", ""))
|
||||
|
||||
block: List[str] = []
|
||||
block.append(f"{index}. [用例 {case['id']}]（对应测试项 {item_id}）：{case.get('test_content', '')}")
|
||||
if title:
|
||||
block.append(f" 场景标题：{title}")
|
||||
block.append(" 操作步骤：")
|
||||
for step_idx, step in enumerate(case.get("operation_steps", []), start=1):
|
||||
block.append(f" {step_idx}) {step}")
|
||||
return block
|
||||
|
||||
def execute(self, context: Dict[str, Any]) -> ToolExecutionResult:
|
||||
test_items = context.get("test_items", {"normal": [], "abnormal": []})
|
||||
test_cases = context.get("test_cases", {"normal": [], "abnormal": []})
|
||||
expected_results = context.get("expected_results", {"normal": [], "abnormal": []})
|
||||
|
||||
lines: List[str] = []
|
||||
|
||||
lines.append("**测试项**")
|
||||
lines.append("")
|
||||
lines.append("**正常测试**：")
|
||||
for index, item in enumerate(test_items.get("normal", []), start=1):
|
||||
lines.append(f"{index}. [测试项 {item['id']}]：{item['content']}")
|
||||
lines.append("")
|
||||
lines.append("**异常测试**：")
|
||||
for index, item in enumerate(test_items.get("abnormal", []), start=1):
|
||||
lines.append(f"{index}. [测试项 {item['id']}]：{item['content']}")
|
||||
|
||||
lines.append("")
|
||||
lines.append("**测试用例**")
|
||||
lines.append("")
|
||||
lines.append("**正常测试**：")
|
||||
for index, case in enumerate(test_cases.get("normal", []), start=1):
|
||||
lines.extend(self._format_case_block(case, index))
|
||||
lines.append("")
|
||||
lines.append("**异常测试**：")
|
||||
for index, case in enumerate(test_cases.get("abnormal", []), start=1):
|
||||
lines.extend(self._format_case_block(case, index))
|
||||
|
||||
lines.append("")
|
||||
lines.append("**预期成果**")
|
||||
lines.append("")
|
||||
lines.append("**正常测试**：")
|
||||
for index, expected in enumerate(expected_results.get("normal", []), start=1):
|
||||
lines.append(
|
||||
f"{index}. [预期 {expected['id']}]（对应用例 {expected['case_id']}）：{expected['result']}"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("**异常测试**：")
|
||||
for index, expected in enumerate(expected_results.get("abnormal", []), start=1):
|
||||
lines.append(
|
||||
f"{index}. [预期 {expected['id']}]（对应用例 {expected['case_id']}）：{expected['result']}"
|
||||
)
|
||||
|
||||
context["formatted_output"] = "\n".join(lines)
|
||||
context["structured_output"] = {
|
||||
"test_items": test_items,
|
||||
"test_cases": test_cases,
|
||||
"expected_results": expected_results,
|
||||
}
|
||||
|
||||
return ToolExecutionResult(
|
||||
context=context,
|
||||
output_summary="formatted_sections=3",
|
||||
)
|
||||
|
||||
|
||||
def build_default_tool_chain() -> List[TestingTool]:
|
||||
return [
|
||||
IdentifyRequirementTypeTool(),
|
||||
DecomposeTestItemsTool(),
|
||||
GenerateTestCasesTool(),
|
||||
BuildExpectedResultsTool(),
|
||||
FormatOutputTool(),
|
||||
]
|
||||
122
rag-web-ui/backend/app/services/vector_schema.py
Normal file
122
rag-web-ui/backend/app/services/vector_schema.py
Normal file
@@ -0,0 +1,122 @@
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChunkVectorMetadata:
|
||||
"""Metadata payload for vector DB and graph linkage."""
|
||||
|
||||
chunk_id: str
|
||||
kb_id: int
|
||||
document_id: int
|
||||
document_name: str
|
||||
document_path: str
|
||||
chunk_index: int
|
||||
chunk_text: str
|
||||
token_count: int
|
||||
language: str = "zh"
|
||||
source_type: str = "document"
|
||||
mission_phase: Optional[str] = None
|
||||
section_title: Optional[str] = None
|
||||
publish_time: Optional[str] = None
|
||||
extracted_entities: List[str] = field(default_factory=list)
|
||||
extracted_entity_types: List[str] = field(default_factory=list)
|
||||
extracted_relations: List[Dict[str, Any]] = field(default_factory=list)
|
||||
graph_node_ids: List[str] = field(default_factory=list)
|
||||
graph_edge_ids: List[str] = field(default_factory=list)
|
||||
community_ids: List[str] = field(default_factory=list)
|
||||
embedding_model: str = ""
|
||||
embedding_dim: int = 0
|
||||
ingest_time: str = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc).isoformat()
|
||||
)
|
||||
|
||||
def to_payload(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"chunk_id": self.chunk_id,
|
||||
"kb_id": self.kb_id,
|
||||
"document_id": self.document_id,
|
||||
"document_name": self.document_name,
|
||||
"document_path": self.document_path,
|
||||
"chunk_index": self.chunk_index,
|
||||
"chunk_text": self.chunk_text,
|
||||
"token_count": self.token_count,
|
||||
"language": self.language,
|
||||
"source_type": self.source_type,
|
||||
"mission_phase": self.mission_phase,
|
||||
"section_title": self.section_title,
|
||||
"publish_time": self.publish_time,
|
||||
"extracted_entities": self.extracted_entities,
|
||||
"extracted_entity_types": self.extracted_entity_types,
|
||||
"extracted_relations": self.extracted_relations,
|
||||
"graph_node_ids": self.graph_node_ids,
|
||||
"graph_edge_ids": self.graph_edge_ids,
|
||||
"community_ids": self.community_ids,
|
||||
"embedding_model": self.embedding_model,
|
||||
"embedding_dim": self.embedding_dim,
|
||||
"ingest_time": self.ingest_time,
|
||||
}
|
||||
|
||||
|
||||
def qdrant_collection_schema(collection_name: str, vector_size: int) -> Dict[str, Any]:
|
||||
"""Qdrant collection and payload index recommendations."""
|
||||
return {
|
||||
"collection_name": collection_name,
|
||||
"vectors": {
|
||||
"size": vector_size,
|
||||
"distance": "Cosine",
|
||||
},
|
||||
"payload_indexes": [
|
||||
{"field_name": "kb_id", "field_schema": "integer"},
|
||||
{"field_name": "document_id", "field_schema": "integer"},
|
||||
{"field_name": "document_name", "field_schema": "keyword"},
|
||||
{"field_name": "chunk_id", "field_schema": "keyword"},
|
||||
{"field_name": "mission_phase", "field_schema": "keyword"},
|
||||
{"field_name": "community_ids", "field_schema": "keyword"},
|
||||
{"field_name": "extracted_entities", "field_schema": "keyword"},
|
||||
{"field_name": "ingest_time", "field_schema": "datetime"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def milvus_collection_schema(collection_name: str, vector_size: int) -> Dict[str, Any]:
|
||||
"""Milvus field design for vector+graph linkage."""
|
||||
return {
|
||||
"collection_name": collection_name,
|
||||
"fields": [
|
||||
{"name": "id", "type": "VARCHAR", "max_length": 64, "is_primary": True},
|
||||
{"name": "kb_id", "type": "INT64"},
|
||||
{"name": "document_id", "type": "INT64"},
|
||||
{"name": "chunk_index", "type": "INT32"},
|
||||
{"name": "document_name", "type": "VARCHAR", "max_length": 255},
|
||||
{"name": "mission_phase", "type": "VARCHAR", "max_length": 64},
|
||||
{"name": "community_ids", "type": "VARCHAR", "max_length": 512},
|
||||
{"name": "extracted_entities", "type": "VARCHAR", "max_length": 2048},
|
||||
{"name": "ingest_time", "type": "VARCHAR", "max_length": 64},
|
||||
{"name": "embedding", "type": "FLOAT_VECTOR", "dim": vector_size},
|
||||
],
|
||||
"index": {
|
||||
"field_name": "embedding",
|
||||
"index_type": "HNSW",
|
||||
"metric_type": "COSINE",
|
||||
"params": {"M": 16, "efConstruction": 200},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
DOCUMENT_CHUNK_METADATA_DDL = """
|
||||
ALTER TABLE document_chunks
|
||||
ADD COLUMN IF NOT EXISTS chunk_index INT NULL,
|
||||
ADD COLUMN IF NOT EXISTS token_count INT NULL,
|
||||
ADD COLUMN IF NOT EXISTS language VARCHAR(16) DEFAULT 'zh',
|
||||
ADD COLUMN IF NOT EXISTS mission_phase VARCHAR(64) NULL,
|
||||
ADD COLUMN IF NOT EXISTS extracted_entities JSON NULL,
|
||||
ADD COLUMN IF NOT EXISTS extracted_entity_types JSON NULL,
|
||||
ADD COLUMN IF NOT EXISTS extracted_relations JSON NULL,
|
||||
ADD COLUMN IF NOT EXISTS graph_node_ids JSON NULL,
|
||||
ADD COLUMN IF NOT EXISTS graph_edge_ids JSON NULL,
|
||||
ADD COLUMN IF NOT EXISTS community_ids JSON NULL,
|
||||
ADD COLUMN IF NOT EXISTS embedding_model VARCHAR(128) NULL,
|
||||
ADD COLUMN IF NOT EXISTS embedding_dim INT NULL;
|
||||
""".strip()
|
||||
11
rag-web-ui/backend/app/services/vector_store/__init__.py
Normal file
11
rag-web-ui/backend/app/services/vector_store/__init__.py
Normal file
@@ -0,0 +1,11 @@
|
||||
from .base import BaseVectorStore
|
||||
from .chroma import ChromaVectorStore
|
||||
from .qdrant import QdrantStore
|
||||
from .factory import VectorStoreFactory
|
||||
|
||||
__all__ = [
|
||||
'BaseVectorStore',
|
||||
'ChromaVectorStore',
|
||||
'QdrantStore',
|
||||
'VectorStoreFactory'
|
||||
]
|
||||
42
rag-web-ui/backend/app/services/vector_store/base.py
Normal file
42
rag-web-ui/backend/app/services/vector_store/base.py
Normal file
@@ -0,0 +1,42 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import List, Optional, Dict, Any
|
||||
from langchain_core.documents import Document
|
||||
from langchain_core.embeddings import Embeddings
|
||||
|
||||
class BaseVectorStore(ABC):
|
||||
"""Abstract base class for vector store implementations"""
|
||||
|
||||
@abstractmethod
|
||||
def __init__(self, collection_name: str, embedding_function: Embeddings, **kwargs):
|
||||
"""Initialize the vector store"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def add_documents(self, documents: List[Document]) -> None:
|
||||
"""Add documents to the vector store"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete(self, ids: List[str]) -> None:
|
||||
"""Delete documents from the vector store"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def as_retriever(self, **kwargs: Any):
|
||||
"""Return a retriever interface for the vector store"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def similarity_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
|
||||
"""Search for similar documents"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def similarity_search_with_score(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
|
||||
"""Search for similar documents with score"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete_collection(self) -> None:
|
||||
"""Delete the entire collection"""
|
||||
pass
|
||||
47
rag-web-ui/backend/app/services/vector_store/chroma.py
Normal file
47
rag-web-ui/backend/app/services/vector_store/chroma.py
Normal file
@@ -0,0 +1,47 @@
|
||||
from typing import List, Any
|
||||
from langchain_core.documents import Document
|
||||
from langchain_core.embeddings import Embeddings
|
||||
from langchain_chroma import Chroma
|
||||
import chromadb
|
||||
from app.core.config import settings
|
||||
|
||||
from .base import BaseVectorStore
|
||||
|
||||
class ChromaVectorStore(BaseVectorStore):
|
||||
"""Chroma vector store implementation"""
|
||||
|
||||
def __init__(self, collection_name: str, embedding_function: Embeddings, **kwargs):
|
||||
"""Initialize Chroma vector store"""
|
||||
chroma_client = chromadb.HttpClient(
|
||||
host=settings.CHROMA_DB_HOST,
|
||||
port=settings.CHROMA_DB_PORT,
|
||||
)
|
||||
|
||||
self._store = Chroma(
|
||||
client=chroma_client,
|
||||
collection_name=collection_name,
|
||||
embedding_function=embedding_function,
|
||||
)
|
||||
def add_documents(self, documents: List[Document]) -> None:
|
||||
"""Add documents to Chroma"""
|
||||
self._store.add_documents(documents)
|
||||
|
||||
def delete(self, ids: List[str]) -> None:
|
||||
"""Delete documents from Chroma"""
|
||||
self._store.delete(ids)
|
||||
|
||||
def as_retriever(self, **kwargs: Any):
|
||||
"""Return a retriever interface"""
|
||||
return self._store.as_retriever(**kwargs)
|
||||
|
||||
def similarity_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
|
||||
"""Search for similar documents in Chroma"""
|
||||
return self._store.similarity_search(query, k=k, **kwargs)
|
||||
|
||||
def similarity_search_with_score(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
|
||||
"""Search for similar documents in Chroma with score"""
|
||||
return self._store.similarity_search_with_score(query, k=k, **kwargs)
|
||||
|
||||
def delete_collection(self) -> None:
|
||||
"""Delete the entire collection"""
|
||||
self._store._client.delete_collection(self._store._collection.name)
|
||||
59
rag-web-ui/backend/app/services/vector_store/factory.py
Normal file
59
rag-web-ui/backend/app/services/vector_store/factory.py
Normal file
@@ -0,0 +1,59 @@
|
||||
from typing import Dict, Type, Any
|
||||
from langchain_core.embeddings import Embeddings
|
||||
|
||||
from .base import BaseVectorStore
|
||||
from .chroma import ChromaVectorStore
|
||||
from .qdrant import QdrantStore
|
||||
|
||||
class VectorStoreFactory:
|
||||
"""Factory for creating vector store instances"""
|
||||
|
||||
_stores: Dict[str, Type[BaseVectorStore]] = {
|
||||
'chroma': ChromaVectorStore,
|
||||
'qdrant': QdrantStore
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
store_type: str,
|
||||
collection_name: str,
|
||||
embedding_function: Embeddings,
|
||||
**kwargs: Any
|
||||
) -> BaseVectorStore:
|
||||
"""Create a vector store instance
|
||||
|
||||
Args:
|
||||
store_type: Type of vector store ('chroma', 'qdrant', etc.)
|
||||
collection_name: Name of the collection
|
||||
embedding_function: Embedding function to use
|
||||
**kwargs: Additional arguments for specific vector store implementations
|
||||
|
||||
Returns:
|
||||
An instance of the requested vector store
|
||||
|
||||
Raises:
|
||||
ValueError: If store_type is not supported
|
||||
"""
|
||||
store_class = cls._stores.get(store_type.lower())
|
||||
if not store_class:
|
||||
raise ValueError(
|
||||
f"Unsupported vector store type: {store_type}. "
|
||||
f"Supported types are: {', '.join(cls._stores.keys())}"
|
||||
)
|
||||
|
||||
return store_class(
|
||||
collection_name=collection_name,
|
||||
embedding_function=embedding_function,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def register_store(cls, name: str, store_class: Type[BaseVectorStore]) -> None:
|
||||
"""Register a new vector store implementation
|
||||
|
||||
Args:
|
||||
name: Name of the vector store type
|
||||
store_class: Vector store class implementation
|
||||
"""
|
||||
cls._stores[name.lower()] = store_class
|
||||
43
rag-web-ui/backend/app/services/vector_store/qdrant.py
Normal file
43
rag-web-ui/backend/app/services/vector_store/qdrant.py
Normal file
@@ -0,0 +1,43 @@
|
||||
from typing import List, Any
|
||||
from langchain_core.documents import Document
|
||||
from langchain_core.embeddings import Embeddings
|
||||
from langchain_community.vectorstores import Qdrant
|
||||
from app.core.config import settings
|
||||
|
||||
from .base import BaseVectorStore
|
||||
|
||||
class QdrantStore(BaseVectorStore):
|
||||
"""Qdrant vector store implementation"""
|
||||
|
||||
def __init__(self, collection_name: str, embedding_function: Embeddings, **kwargs):
|
||||
"""Initialize Qdrant vector store"""
|
||||
self._store = Qdrant(
|
||||
collection_name=collection_name,
|
||||
embeddings=embedding_function,
|
||||
url=settings.QDRANT_URL,
|
||||
prefer_grpc=settings.QDRANT_PREFER_GRPC
|
||||
)
|
||||
|
||||
def add_documents(self, documents: List[Document]) -> None:
|
||||
"""Add documents to Qdrant"""
|
||||
self._store.add_documents(documents)
|
||||
|
||||
def delete(self, ids: List[str]) -> None:
|
||||
"""Delete documents from Qdrant"""
|
||||
self._store.delete(ids)
|
||||
|
||||
def as_retriever(self, **kwargs: Any):
|
||||
"""Return a retriever interface"""
|
||||
return self._store.as_retriever(**kwargs)
|
||||
|
||||
def similarity_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
|
||||
"""Search for similar documents in Qdrant"""
|
||||
return self._store.similarity_search(query, k=k, **kwargs)
|
||||
|
||||
def similarity_search_with_score(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
|
||||
"""Search for similar documents in Qdrant with score"""
|
||||
return self._store.similarity_search_with_score(query, k=k, **kwargs)
|
||||
|
||||
def delete_collection(self) -> None:
|
||||
"""Delete the entire collection"""
|
||||
self._store._client.delete_collection(self._store._collection_name)
|
||||
100
rag-web-ui/backend/app/startup/migarate.py
Normal file
100
rag-web-ui/backend/app/startup/migarate.py
Normal file
@@ -0,0 +1,100 @@
|
||||
import logging
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Generator, Tuple
|
||||
|
||||
from alembic.config import Config
|
||||
from alembic.config import main as alembic_main
|
||||
from alembic.migration import MigrationContext
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.engine import Connection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DatabaseMigrator:
|
||||
"""
|
||||
Database migrator class
|
||||
"""
|
||||
|
||||
def __init__(self, db_url: str):
|
||||
self.db_url = db_url
|
||||
self.alembic_cfg = self._get_alembic_config()
|
||||
|
||||
@contextmanager
|
||||
def database_connection(self) -> Generator[Connection, None, None]:
|
||||
"""
|
||||
Context manager for database connections with timeout
|
||||
|
||||
Yields:
|
||||
SQLAlchemy connection object
|
||||
"""
|
||||
engine = create_engine(
|
||||
self.db_url, connect_args={"connect_timeout": 3} # 设置连接超时为3秒
|
||||
)
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
yield connection
|
||||
except Exception as e:
|
||||
logger.error(f"Database connection error: {e}")
|
||||
raise
|
||||
|
||||
def check_migration_needed(self) -> Tuple[bool, str, str]:
|
||||
"""
|
||||
Check if database migration is needed
|
||||
|
||||
Returns:
|
||||
Tuple containing:
|
||||
- bool: Whether migration is needed
|
||||
- str: Current revision
|
||||
- str: Head revision
|
||||
"""
|
||||
with self.database_connection() as connection:
|
||||
context = MigrationContext.configure(connection)
|
||||
current_rev = context.get_current_revision()
|
||||
heads = context.get_current_heads()
|
||||
|
||||
if not heads:
|
||||
logger.warning("No migration heads found. Database might not be initialized.")
|
||||
return True, current_rev or "None", "head"
|
||||
|
||||
head_rev = heads[0]
|
||||
return current_rev != head_rev, current_rev or "None", head_rev
|
||||
|
||||
def _get_alembic_config(self) -> Config:
|
||||
"""
|
||||
Create and configure Alembic config
|
||||
|
||||
Returns:
|
||||
Alembic config object
|
||||
"""
|
||||
project_root = Path(__file__).resolve().parents[2] # Go up 3 levels from migrate.py
|
||||
alembic_cfg = Config(project_root / "alembic.ini")
|
||||
alembic_cfg.set_main_option("sqlalchemy.url", self.db_url)
|
||||
return alembic_cfg
|
||||
|
||||
def run_migrations(self) -> None:
|
||||
"""
|
||||
Run database migrations if needed
|
||||
|
||||
Raises:
|
||||
Exception: If migration fails
|
||||
"""
|
||||
try:
|
||||
# Check if migration is needed
|
||||
needs_migration, current_rev, head_rev = self.check_migration_needed()
|
||||
|
||||
if needs_migration:
|
||||
logger.info(f"Current revision: {current_rev}, upgrading to: {head_rev}")
|
||||
self.alembic_cfg.set_main_option("sqlalchemy.url", self.db_url)
|
||||
|
||||
# 执行 alembic 升级
|
||||
alembic_main(argv=["--raiseerr", "upgrade", "head"], config=self.alembic_cfg)
|
||||
|
||||
logger.info("Database migrations completed successfully")
|
||||
else:
|
||||
logger.info(f"Database is already at the latest version: {current_rev}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error during database migration: {e}")
|
||||
raise
|
||||
4
rag-web-ui/backend/app/tools/__init__.py
Normal file
4
rag-web-ui/backend/app/tools/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
from app.tools.base import ToolDefinition
|
||||
from app.tools.registry import ToolRegistry
|
||||
|
||||
__all__ = ["ToolDefinition", "ToolRegistry"]
|
||||
11
rag-web-ui/backend/app/tools/base.py
Normal file
11
rag-web-ui/backend/app/tools/base.py
Normal file
@@ -0,0 +1,11 @@
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ToolDefinition:
|
||||
name: str
|
||||
version: str
|
||||
description: str
|
||||
input_schema: Dict[str, Any]
|
||||
output_schema: Dict[str, Any]
|
||||
19
rag-web-ui/backend/app/tools/registry.py
Normal file
19
rag-web-ui/backend/app/tools/registry.py
Normal file
@@ -0,0 +1,19 @@
|
||||
from typing import Dict, List
|
||||
|
||||
from app.tools.base import ToolDefinition
|
||||
|
||||
|
||||
class ToolRegistry:
|
||||
_tools: Dict[str, ToolDefinition] = {}
|
||||
|
||||
@classmethod
|
||||
def register(cls, definition: ToolDefinition) -> None:
|
||||
cls._tools[definition.name] = definition
|
||||
|
||||
@classmethod
|
||||
def get(cls, name: str) -> ToolDefinition:
|
||||
return cls._tools[name]
|
||||
|
||||
@classmethod
|
||||
def list(cls) -> List[ToolDefinition]:
|
||||
return list(cls._tools.values())
|
||||
3
rag-web-ui/backend/app/tools/srs_reqs_qwen/__init__.py
Normal file
3
rag-web-ui/backend/app/tools/srs_reqs_qwen/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from app.tools.srs_reqs_qwen.tool import SRSTool, get_srs_tool
|
||||
|
||||
__all__ = ["SRSTool", "get_srs_tool"]
|
||||
102
rag-web-ui/backend/app/tools/srs_reqs_qwen/default_config.yaml
Normal file
102
rag-web-ui/backend/app/tools/srs_reqs_qwen/default_config.yaml
Normal file
@@ -0,0 +1,102 @@
|
||||
# 配置文件 - SRS 需求文档解析工具 (LLM增强版)
|
||||
# Configuration file for SRS Requirement Document Parser (LLM Enhanced Version)
|
||||
|
||||
# LLM配置 - 阿里云千问
|
||||
llm:
|
||||
# 是否启用LLM（设为false则使用纯规则提取）
|
||||
enabled: true
|
||||
# LLM提供商：qwen（阿里云千问）
|
||||
provider: "qwen"
|
||||
# 模型名称
|
||||
model: "qwen3-max"
|
||||
# API密钥统一由 rag-web-ui 的环境变量提供
|
||||
api_key: ""
|
||||
# 可选参数
|
||||
temperature: 0.3
|
||||
max_tokens: 1024
|
||||
|
||||
# 文档解析配置
|
||||
document:
|
||||
supported_formats:
|
||||
- ".pdf"
|
||||
- ".docx"
|
||||
# 标题识别的样式列表
|
||||
heading_styles:
|
||||
- "Heading 1"
|
||||
- "Heading 2"
|
||||
- "Heading 3"
|
||||
- "Heading 4"
|
||||
- "Heading 5"
|
||||
# 需要过滤的非需求章节（GJB438B标准）
|
||||
non_requirement_sections:
|
||||
- "标识"
|
||||
- "系统概述"
|
||||
- "文档概述"
|
||||
- "引用文档"
|
||||
- "合格性规定"
|
||||
- "需求可追踪性"
|
||||
- "注释"
|
||||
- "附录"
|
||||
|
||||
# 需求提取配置
|
||||
extraction:
|
||||
# 需求类型关键字（用于自动判断需求类型）
|
||||
requirement_types:
|
||||
功能需求:
|
||||
prefix: "FR"
|
||||
keywords: ["功能", "feature", "requirement", "CSCI组成", "控制", "处理", "监测", "显示"]
|
||||
priority: 1
|
||||
接口需求:
|
||||
prefix: "IR"
|
||||
keywords: ["接口", "interface", "api", "外部接口", "内部接口", "CAN", "以太网", "通信"]
|
||||
priority: 2
|
||||
性能需求:
|
||||
prefix: "PR"
|
||||
keywords: ["性能", "performance", "速度", "响应时间", "吞吐量"]
|
||||
priority: 3
|
||||
安全需求:
|
||||
prefix: "SR"
|
||||
keywords: ["安全", "security", "安全性", "报警"]
|
||||
priority: 4
|
||||
可靠性需求:
|
||||
prefix: "RR"
|
||||
keywords: ["可靠", "reliability", "容错", "恢复", "冗余"]
|
||||
priority: 5
|
||||
其他需求:
|
||||
prefix: "OR"
|
||||
keywords: ["约束", "资源", "适应性", "保密", "环境", "计算机", "质量", "设计", "人员", "培训", "保障", "验收", "交付"]
|
||||
priority: 6
|
||||
splitter:
|
||||
enabled: true
|
||||
max_sentence_len: 120
|
||||
min_clause_len: 12
|
||||
semantic_guard:
|
||||
enabled: true
|
||||
preserve_condition_action_chain: true
|
||||
preserve_alarm_chain: true
|
||||
table_strategy:
|
||||
llm_semantic_enabled: true
|
||||
sequence_table_merge: "single_requirement"
|
||||
merge_time_series_rows_min: 3
|
||||
rewrite_policy:
|
||||
llm_light_rewrite_enabled: true
|
||||
preserve_ratio_min: 0.65
|
||||
max_length_growth_ratio: 1.25
|
||||
renumber_policy:
|
||||
enabled: true
|
||||
mode: "section_continuous"
|
||||
|
||||
# 输出配置
|
||||
output:
|
||||
format: "json"
|
||||
indent: 2
|
||||
# 是否美化输出（格式化）
|
||||
pretty_print: true
|
||||
# 是否包含元数据
|
||||
include_metadata: true
|
||||
|
||||
# 日志配置
|
||||
logging:
|
||||
level: "INFO" # DEBUG, INFO, WARNING, ERROR
|
||||
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
file: "srs_parser.log"
|
||||
26
rag-web-ui/backend/app/tools/srs_reqs_qwen/src/__init__.py
Normal file
26
rag-web-ui/backend/app/tools/srs_reqs_qwen/src/__init__.py
Normal file
@@ -0,0 +1,26 @@
|
||||
# src/__init__.py
|
||||
"""
|
||||
SRS 需求文档解析工具包
|
||||
"""
|
||||
|
||||
__version__ = "1.0.0"
|
||||
__author__ = "SRS Parser Team"
|
||||
|
||||
from .document_parser import DocumentParser
|
||||
from .llm_interface import LLMInterface, QwenLLM
|
||||
from .requirement_extractor import RequirementExtractor
|
||||
from .json_generator import JSONGenerator
|
||||
from .settings import AppSettings
|
||||
from .requirement_splitter import RequirementSplitter
|
||||
from .requirement_id_generator import RequirementIDGenerator
|
||||
|
||||
__all__ = [
|
||||
'DocumentParser',
|
||||
'LLMInterface',
|
||||
'QwenLLM',
|
||||
'RequirementExtractor',
|
||||
'JSONGenerator',
|
||||
'AppSettings',
|
||||
'RequirementSplitter',
|
||||
'RequirementIDGenerator',
|
||||
]
|
||||
@@ -0,0 +1,709 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
文档解析模块 - LLM增强版
|
||||
支持PDF和Docx格式，针对GJB438B标准SRS文档优化
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import logging
|
||||
import importlib
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import List, Dict, Tuple, Optional, Any
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from docx import Document
|
||||
HAS_DOCX = True
|
||||
except ImportError:
|
||||
HAS_DOCX = False
|
||||
|
||||
try:
|
||||
import PyPDF2
|
||||
HAS_PDF = True
|
||||
except ImportError:
|
||||
HAS_PDF = False
|
||||
|
||||
HAS_PDF_TABLE = importlib.util.find_spec("pdfplumber") is not None
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Section:
|
||||
"""表示文档中的一个章节"""
|
||||
|
||||
def __init__(self, level: int, title: str, number: str = None, content: str = "", uid: str = ""):
|
||||
self.level = level
|
||||
self.title = title
|
||||
self.number = number
|
||||
self.content = content
|
||||
self.uid = uid
|
||||
self.parent = None
|
||||
self.children = []
|
||||
self.tables = []
|
||||
self.blocks = []
|
||||
|
||||
def add_child(self, child: 'Section') -> None:
|
||||
self.children.append(child)
|
||||
child.parent = self
|
||||
|
||||
def add_content(self, text: str) -> None:
|
||||
text = (text or "").strip()
|
||||
if not text:
|
||||
return
|
||||
if self.content:
|
||||
self.content += "\n" + text
|
||||
else:
|
||||
self.content = text
|
||||
self.blocks.append({"type": "text", "text": text})
|
||||
|
||||
def add_table(self, table_data: List[List[str]]) -> None:
|
||||
if not table_data:
|
||||
return
|
||||
self.tables.append(table_data)
|
||||
table_index = len(self.tables) - 1
|
||||
self.blocks.append({"type": "table", "table_index": table_index, "table": table_data})
|
||||
|
||||
def generate_auto_number(self, parent_number: str = "", sibling_index: int = 1) -> None:
|
||||
"""
|
||||
自动生成章节编号（当章节没有编号时）
|
||||
|
||||
Args:
|
||||
parent_number: 父章节编号
|
||||
sibling_index: 在同级章节中的序号（从1开始）
|
||||
"""
|
||||
if not self.number:
|
||||
if parent_number:
|
||||
self.number = f"{parent_number}.{sibling_index}"
|
||||
else:
|
||||
self.number = str(sibling_index)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Section(level={self.level}, number='{self.number}', title='{self.title}')"
|
||||
|
||||
|
||||
class DocumentParser(ABC):
|
||||
"""文档解析器基类"""
|
||||
|
||||
def __init__(self, file_path: str):
|
||||
self.file_path = file_path
|
||||
self.sections: List[Section] = []
|
||||
self.document_title = ""
|
||||
self.raw_text = ""
|
||||
self.llm = None
|
||||
self._uid_counter = 0
|
||||
|
||||
def set_llm(self, llm) -> None:
|
||||
"""设置LLM实例"""
|
||||
self.llm = llm
|
||||
|
||||
@abstractmethod
|
||||
def parse(self) -> List[Section]:
|
||||
pass
|
||||
|
||||
def get_document_title(self) -> str:
|
||||
return self.document_title
|
||||
|
||||
def _next_uid(self) -> str:
|
||||
self._uid_counter += 1
|
||||
return f"sec-{self._uid_counter}"
|
||||
|
||||
def _auto_number_sections(self, sections: List[Section], parent_number: str = "") -> None:
|
||||
"""
|
||||
为没有编号的章节自动生成编号
|
||||
|
||||
规则：使用Word样式确定级别，跳过前置章节（目录、概述等），
|
||||
从第一个正文章节（如"外部接口"）开始编号为1
|
||||
|
||||
Args:
|
||||
sections: 章节列表
|
||||
parent_number: 父章节编号
|
||||
"""
|
||||
# 仅在顶级章节重编号
|
||||
if not parent_number:
|
||||
# 前置章节关键词（需要跳过的）
|
||||
skip_keywords = ['目录', '封面', '扉页', '未命名', '年', '月']
|
||||
# 正文章节关键词（遇到这些说明正文开始）
|
||||
content_keywords = ['外部接口', '接口', '软件需求', '需求', '功能', '性能', '设计', '概述', '标识', '引言']
|
||||
|
||||
start_index = 0
|
||||
for idx, section in enumerate(sections):
|
||||
# 优先检查是否是正文章节
|
||||
is_content = any(kw in section.title for kw in content_keywords)
|
||||
if is_content and section.level == 1:
|
||||
start_index = idx
|
||||
break
|
||||
|
||||
# 重新编号所有章节
|
||||
counter = 1
|
||||
for i, section in enumerate(sections):
|
||||
if i < start_index:
|
||||
# 前置章节不编号
|
||||
section.number = ""
|
||||
else:
|
||||
# 正文章节：顶级章节从1开始编号
|
||||
if section.level == 1:
|
||||
section.number = str(counter)
|
||||
counter += 1
|
||||
|
||||
# 递归处理子章节
|
||||
if section.children:
|
||||
self._auto_number_sections(section.children, section.number)
|
||||
else:
|
||||
# 子章节编号
|
||||
for i, section in enumerate(sections, 1):
|
||||
if not section.number or self._is_chinese_number(section.number):
|
||||
section.generate_auto_number(parent_number, i)
|
||||
if section.children:
|
||||
self._auto_number_sections(section.children, section.number)
|
||||
|
||||
def _is_chinese_number(self, text: str) -> bool:
|
||||
"""检查是否是中文数字编号"""
|
||||
chinese_numbers = '一二三四五六七八九十百千万'
|
||||
return text and all(c in chinese_numbers for c in text)
|
||||
|
||||
|
||||
class DocxParser(DocumentParser):
|
||||
"""DOCX格式文档解析器"""
|
||||
|
||||
def __init__(self, file_path: str):
|
||||
if not HAS_DOCX:
|
||||
raise ImportError("python-docx库未安装，请运行: pip install python-docx")
|
||||
super().__init__(file_path)
|
||||
self.document = None
|
||||
|
||||
def parse(self) -> List[Section]:
|
||||
try:
|
||||
self.document = Document(self.file_path)
|
||||
self.document_title = self.document.core_properties.title or "SRS Document"
|
||||
|
||||
section_stack = {}
|
||||
|
||||
for block in self._iter_block_items(self.document):
|
||||
from docx.text.paragraph import Paragraph
|
||||
from docx.table import Table
|
||||
if isinstance(block, Paragraph):
|
||||
text = block.text.strip()
|
||||
if not text:
|
||||
continue
|
||||
|
||||
heading_info = self._parse_heading(block, text)
|
||||
if heading_info:
|
||||
number, title, level = heading_info
|
||||
section = Section(level=level, title=title, number=number, uid=self._next_uid())
|
||||
|
||||
if level == 1 or not section_stack:
|
||||
self.sections.append(section)
|
||||
section_stack = {1: section}
|
||||
else:
|
||||
parent_level = level - 1
|
||||
while parent_level >= 1 and parent_level not in section_stack:
|
||||
parent_level -= 1
|
||||
|
||||
if parent_level >= 1 and parent_level in section_stack:
|
||||
section_stack[parent_level].add_child(section)
|
||||
elif self.sections:
|
||||
self.sections[-1].add_child(section)
|
||||
|
||||
section_stack[level] = section
|
||||
for l in list(section_stack.keys()):
|
||||
if l > level:
|
||||
del section_stack[l]
|
||||
else:
|
||||
# 添加内容到当前章节
|
||||
if section_stack:
|
||||
max_level = max(section_stack.keys())
|
||||
section_stack[max_level].add_content(text)
|
||||
else:
|
||||
# 没有标题时，创建默认章节
|
||||
default_section = Section(level=1, title="未命名章节", number="", uid=self._next_uid())
|
||||
default_section.add_content(text)
|
||||
self.sections.append(default_section)
|
||||
section_stack = {1: default_section}
|
||||
elif isinstance(block, Table):
|
||||
# 表格处理
|
||||
table_data = self._extract_table_data(block)
|
||||
if table_data:
|
||||
if section_stack:
|
||||
max_level = max(section_stack.keys())
|
||||
section_stack[max_level].add_table(table_data)
|
||||
else:
|
||||
default_section = Section(level=1, title="未命名章节", number="", uid=self._next_uid())
|
||||
default_section.add_table(table_data)
|
||||
self.sections.append(default_section)
|
||||
section_stack = {1: default_section}
|
||||
|
||||
# 为没有编号的章节自动生成编号
|
||||
self._auto_number_sections(self.sections)
|
||||
|
||||
logger.info(f"完成Docx解析，提取{len(self.sections)}个顶级章节")
|
||||
return self.sections
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"解析Docx文档失败: {e}")
|
||||
raise
|
||||
|
||||
def _is_valid_heading(self, text: str) -> bool:
|
||||
"""检查是否是有效的标题"""
|
||||
if len(text) > 120 or '...' in text:
|
||||
return False
|
||||
# 标题应包含中文或字母
|
||||
if not re.search(r'[\u4e00-\u9fa5A-Za-z]', text):
|
||||
return False
|
||||
# 过滤目录项（标题后跟页码，如"概述 2"或"概述 . . . . 2"）
|
||||
if re.search(r'\s{2,}\d+$', text): # 多个空格后跟数字结尾
|
||||
return False
|
||||
if re.search(r'[\.。\s]+\d+$', text): # 点号或空格后跟数字结尾
|
||||
return False
|
||||
return True
|
||||
|
||||
def _parse_heading(self, paragraph, text: str) -> Optional[Tuple[str, str, int]]:
|
||||
"""解析标题，返回(编号, 标题, 级别)"""
|
||||
style_name = paragraph.style.name if paragraph.style else ""
|
||||
is_heading_style = style_name.lower().startswith('heading') if style_name else False
|
||||
|
||||
# 数字编号标题
|
||||
match = re.match(r'^(\d+(?:\.\d+)*)\s*[\.、]?\s*(.+)$', text)
|
||||
if match and self._is_valid_heading(match.group(2)):
|
||||
number = match.group(1)
|
||||
title = match.group(2).strip()
|
||||
level = len(number.split('.'))
|
||||
return number, title, level
|
||||
|
||||
# 中文编号标题
|
||||
match = re.match(r'^([一二三四五六七八九十]+)[、\.]+\s*(.+)$', text)
|
||||
if match and self._is_valid_heading(match.group(2)):
|
||||
number = match.group(1)
|
||||
title = match.group(2).strip()
|
||||
level = 1
|
||||
return number, title, level
|
||||
|
||||
# 样式标题
|
||||
if is_heading_style and self._is_valid_heading(text):
|
||||
level = 1
|
||||
level_match = re.search(r'(\d+)', style_name)
|
||||
if level_match:
|
||||
level = int(level_match.group(1))
|
||||
return "", text, level
|
||||
|
||||
return None
|
||||
|
||||
def _iter_block_items(self, parent):
|
||||
"""按文档顺序迭代段落和表格"""
|
||||
from docx.text.paragraph import Paragraph
|
||||
from docx.table import Table
|
||||
from docx.oxml.text.paragraph import CT_P
|
||||
from docx.oxml.table import CT_Tbl
|
||||
|
||||
for child in parent.element.body.iterchildren():
|
||||
if isinstance(child, CT_P):
|
||||
yield Paragraph(child, parent)
|
||||
elif isinstance(child, CT_Tbl):
|
||||
yield Table(child, parent)
|
||||
|
||||
def _extract_table_data(self, table) -> List[List[str]]:
|
||||
"""提取表格数据"""
|
||||
table_data = []
|
||||
for row in table.rows:
|
||||
row_data = []
|
||||
for cell in row.cells:
|
||||
text = cell.text.replace('\n', ' ').strip()
|
||||
text = re.sub(r'\s+', ' ', text)
|
||||
row_data.append(text)
|
||||
if any(cell for cell in row_data):
|
||||
table_data.append(row_data)
|
||||
return table_data
|
||||
|
||||
|
||||
class PDFParser(DocumentParser):
|
||||
"""PDF格式文档解析器 - LLM增强版"""
|
||||
|
||||
# GJB438B标准SRS文档的有效章节标题关键词
|
||||
VALID_TITLE_KEYWORDS = [
|
||||
'范围', '标识', '概述', '引用', '文档',
|
||||
'需求', '功能', '接口', '性能', '安全', '保密',
|
||||
'环境', '资源', '质量', '设计', '约束',
|
||||
'人员', '培训', '保障', '验收', '交付', '包装',
|
||||
'优先', '关键', '合格', '追踪', '注释',
|
||||
'CSCI', '计算机', '软件', '硬件', '通信', '通讯',
|
||||
'数据', '适应', '可靠', '内部', '外部',
|
||||
'描述', '要求', '规定', '说明', '定义',
|
||||
'电场', '防护', '装置', '控制', '监控', '显控'
|
||||
]
|
||||
|
||||
# 明显无效的章节标题模式（噪声）
|
||||
INVALID_TITLE_PATTERNS = [
|
||||
'本文档可作为', '参比电位', '补偿电流', '以太网',
|
||||
'电源', '软件接', '功能\\', '性能 \\', '输入/输出 \\',
|
||||
'数据处理要求 \\', '固件 \\', '质量控制要求',
|
||||
'信安科技', '浙江', '公司'
|
||||
]
|
||||
|
||||
def __init__(self, file_path: str):
|
||||
if not HAS_PDF:
|
||||
raise ImportError("PyPDF2库未安装，请运行: pip install PyPDF2")
|
||||
super().__init__(file_path)
|
||||
self.document_title = "SRS Document"
|
||||
self._page_texts: List[str] = []
|
||||
|
||||
def parse(self) -> List[Section]:
|
||||
"""解析PDF文档"""
|
||||
try:
|
||||
# 1. 提取所有文本
|
||||
self.raw_text = self._extract_all_text()
|
||||
|
||||
# 2. 清洗文本
|
||||
cleaned_text = self._clean_text(self.raw_text)
|
||||
|
||||
# 3. 识别章节结构
|
||||
self.sections = self._parse_sections(cleaned_text)
|
||||
|
||||
# 4. 使用LLM验证和清理章节（如果可用）
|
||||
if self.llm:
|
||||
self.sections = self._llm_validate_sections(self.sections)
|
||||
|
||||
# 章节识别失败时，创建兜底章节避免后续表格数据丢失。
|
||||
if not self.sections:
|
||||
fallback = Section(level=1, title="未命名章节", number="1", uid=self._next_uid())
|
||||
if cleaned_text:
|
||||
fallback.add_content(cleaned_text)
|
||||
self.sections = [fallback]
|
||||
|
||||
# 5. 提取并挂接PDF表格到章节（若依赖可用）
|
||||
pdf_tables = self._extract_pdf_tables()
|
||||
if pdf_tables:
|
||||
self._attach_pdf_tables_to_sections(pdf_tables)
|
||||
|
||||
# 6. 为没有编号的章节自动生成编号
|
||||
self._auto_number_sections(self.sections)
|
||||
|
||||
logger.info(f"完成PDF解析，提取{len(self.sections)}个顶级章节")
|
||||
return self.sections
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"解析PDF文档失败: {e}")
|
||||
raise
|
||||
|
||||
def _extract_all_text(self) -> str:
|
||||
"""从PDF提取所有文本"""
|
||||
all_text = []
|
||||
with open(self.file_path, 'rb') as f:
|
||||
pdf_reader = PyPDF2.PdfReader(f)
|
||||
for page in pdf_reader.pages:
|
||||
text = page.extract_text()
|
||||
if text:
|
||||
all_text.append(text)
|
||||
self._page_texts = all_text
|
||||
return '\n'.join(all_text)
|
||||
|
||||
def _extract_pdf_tables(self) -> List[Dict[str, Any]]:
|
||||
"""提取PDF中的表格数据。"""
|
||||
if not HAS_PDF_TABLE:
|
||||
logger.warning("未安装pdfplumber，跳过PDF表格提取。可执行: pip install pdfplumber")
|
||||
return []
|
||||
|
||||
tables: List[Dict[str, Any]] = []
|
||||
try:
|
||||
pdfplumber = importlib.import_module("pdfplumber")
|
||||
with pdfplumber.open(self.file_path) as pdf:
|
||||
for page_idx, page in enumerate(pdf.pages):
|
||||
page_text = ""
|
||||
if page_idx < len(self._page_texts):
|
||||
page_text = self._page_texts[page_idx]
|
||||
|
||||
extracted_tables = page.extract_tables() or []
|
||||
for table_idx, table in enumerate(extracted_tables):
|
||||
cleaned_table: List[List[str]] = []
|
||||
for row in table or []:
|
||||
cells = [re.sub(r'\s+', ' ', str(cell or '')).strip() for cell in row]
|
||||
if any(cells):
|
||||
cleaned_table.append(cells)
|
||||
|
||||
if cleaned_table:
|
||||
tables.append(
|
||||
{
|
||||
"page_idx": page_idx,
|
||||
"table_idx": table_idx,
|
||||
"page_text": page_text,
|
||||
"data": cleaned_table,
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"PDF表格提取失败，继续纯文本流程: {e}")
|
||||
return []
|
||||
|
||||
logger.info(f"PDF表格提取完成，共{len(tables)}个表格")
|
||||
return tables
|
||||
|
||||
def _attach_pdf_tables_to_sections(self, tables: List[Dict[str, Any]]) -> None:
|
||||
"""将提取出的PDF表格挂接到最匹配的章节。"""
|
||||
flat_sections = self._flatten_sections(self.sections)
|
||||
if not flat_sections:
|
||||
return
|
||||
|
||||
last_section: Optional[Section] = None
|
||||
for table in tables:
|
||||
matched = self._match_table_section(table.get("page_text", ""), flat_sections)
|
||||
target = matched or last_section or flat_sections[0]
|
||||
target.add_table(table["data"])
|
||||
last_section = target
|
||||
|
||||
def _flatten_sections(self, sections: List[Section]) -> List[Section]:
|
||||
"""按文档顺序拉平章节树。"""
|
||||
result: List[Section] = []
|
||||
for section in sections:
|
||||
result.append(section)
|
||||
if section.children:
|
||||
result.extend(self._flatten_sections(section.children))
|
||||
return result
|
||||
|
||||
def _match_table_section(self, page_text: str, sections: List[Section]) -> Optional[Section]:
|
||||
"""基于页文本匹配表格归属章节。"""
|
||||
normalized_page = re.sub(r"\s+", "", (page_text or "")).lower()
|
||||
if not normalized_page:
|
||||
return None
|
||||
|
||||
matched: Optional[Section] = None
|
||||
matched_score = -1
|
||||
for section in sections:
|
||||
title = (section.title or "").strip()
|
||||
if not title:
|
||||
continue
|
||||
|
||||
number = (section.number or "").strip()
|
||||
candidates = [title]
|
||||
if number:
|
||||
candidates.append(f"{number}{title}")
|
||||
candidates.append(f"{number} {title}")
|
||||
|
||||
for candidate in candidates:
|
||||
normalized_candidate = re.sub(r"\s+", "", candidate).lower()
|
||||
if normalized_candidate and normalized_candidate in normalized_page:
|
||||
score = len(normalized_candidate)
|
||||
if score > matched_score:
|
||||
matched = section
|
||||
matched_score = score
|
||||
|
||||
return matched
|
||||
|
||||
def _clean_text(self, text: str) -> str:
|
||||
"""清洗PDF提取的文本"""
|
||||
lines = text.split('\n')
|
||||
cleaned_lines = []
|
||||
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
# 跳过页码（通常是1-3位数字单独一行）
|
||||
if re.match(r'^\d{1,3}$', line):
|
||||
continue
|
||||
# 跳过目录行
|
||||
if line.count('.') > 10 and '...' in line:
|
||||
continue
|
||||
|
||||
cleaned_lines.append(line)
|
||||
|
||||
return '\n'.join(cleaned_lines)
|
||||
|
||||
def _parse_sections(self, text: str) -> List[Section]:
|
||||
"""解析章节结构"""
|
||||
sections = []
|
||||
section_stack = {}
|
||||
lines = text.split('\n')
|
||||
current_section = None
|
||||
content_buffer = []
|
||||
found_sections = set()
|
||||
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
# 尝试匹配章节标题
|
||||
section_info = self._match_section_header(line, found_sections)
|
||||
|
||||
if section_info:
|
||||
number, title = section_info
|
||||
level = len(number.split('.'))
|
||||
|
||||
# 保存之前章节的内容
|
||||
if current_section and content_buffer:
|
||||
current_section.add_content('\n'.join(content_buffer))
|
||||
content_buffer = []
|
||||
|
||||
# 创建新章节
|
||||
section = Section(level=level, title=title, number=number, uid=self._next_uid())
|
||||
found_sections.add(number)
|
||||
|
||||
# 建立层次结构
|
||||
if level == 1:
|
||||
sections.append(section)
|
||||
section_stack = {1: section}
|
||||
else:
|
||||
parent_level = level - 1
|
||||
while parent_level >= 1 and parent_level not in section_stack:
|
||||
parent_level -= 1
|
||||
|
||||
if parent_level >= 1 and parent_level in section_stack:
|
||||
section_stack[parent_level].add_child(section)
|
||||
elif sections:
|
||||
sections[-1].add_child(section)
|
||||
else:
|
||||
sections.append(section)
|
||||
section_stack = {1: section}
|
||||
|
||||
section_stack[level] = section
|
||||
for l in list(section_stack.keys()):
|
||||
if l > level:
|
||||
del section_stack[l]
|
||||
|
||||
current_section = section
|
||||
else:
|
||||
# 收集内容
|
||||
if line and not self._is_noise(line):
|
||||
content_buffer.append(line)
|
||||
|
||||
# 保存最后一个章节的内容
|
||||
if current_section and content_buffer:
|
||||
current_section.add_content('\n'.join(content_buffer))
|
||||
|
||||
return sections
|
||||
|
||||
def _match_section_header(self, line: str, found_sections: set) -> Optional[Tuple[str, str]]:
|
||||
"""
|
||||
匹配章节标题
|
||||
|
||||
Returns:
|
||||
(章节编号, 章节标题) 或 None
|
||||
"""
|
||||
# 模式: "3.1功能需求" 或 "3.1 功能需求"
|
||||
match = re.match(r'^(\d+(?:\.\d+)*)\s*(.+)$', line)
|
||||
if not match:
|
||||
return None
|
||||
|
||||
number = match.group(1)
|
||||
title = match.group(2).strip()
|
||||
|
||||
# 排除目录行
|
||||
if '...' in title or title.count('.') > 5:
|
||||
return None
|
||||
|
||||
# 验证章节编号
|
||||
parts = number.split('.')
|
||||
first_part = int(parts[0])
|
||||
|
||||
# 放宽一级章节编号范围（非严格GJB结构）
|
||||
if first_part < 1 or first_part > 30:
|
||||
return None
|
||||
|
||||
# 检查子部分是否合理
|
||||
for part in parts[1:]:
|
||||
if int(part) > 20:
|
||||
return None
|
||||
|
||||
# 避免重复
|
||||
if number in found_sections:
|
||||
return None
|
||||
|
||||
# 标题长度检查
|
||||
if len(title) > 60 or len(title) < 2:
|
||||
return None
|
||||
|
||||
# 放宽标题字符要求（兼容部分PDF字体导致中文抽取异常的情况）
|
||||
if not re.search(r'[\u4e00-\u9fa5A-Za-z]', title):
|
||||
return None
|
||||
|
||||
# 检查是否包含无效模式
|
||||
for invalid_pattern in self.INVALID_TITLE_PATTERNS:
|
||||
if invalid_pattern in title:
|
||||
return None
|
||||
|
||||
# 标题不能以数字开头
|
||||
if title[0].isdigit():
|
||||
return None
|
||||
|
||||
# 数字比例检查
|
||||
digit_ratio = sum(c.isdigit() for c in title) / max(len(title), 1)
|
||||
if digit_ratio > 0.3:
|
||||
return None
|
||||
|
||||
# 检查标题是否包含反斜杠（通常是表格噪声）
|
||||
if '\\' in title and '需求' not in title:
|
||||
return None
|
||||
|
||||
return (number, title)
|
||||
|
||||
def _is_noise(self, line: str) -> bool:
|
||||
"""检查是否是噪声行"""
|
||||
# 纯数字行
|
||||
if re.match(r'^[\d\s,.]+$', line):
|
||||
return True
|
||||
# 非常短的行
|
||||
if len(line) < 3:
|
||||
return True
|
||||
# 罗马数字
|
||||
if re.match(r'^[ivxIVX]+$', line):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _llm_validate_sections(self, sections: List[Section]) -> List[Section]:
|
||||
"""使用LLM验证章节是否有效"""
|
||||
if not self.llm:
|
||||
return sections
|
||||
|
||||
validated_sections = []
|
||||
|
||||
for section in sections:
|
||||
# 验证顶级章节
|
||||
if self._is_valid_section_with_llm(section):
|
||||
# 递归验证子章节
|
||||
section.children = self._validate_children(section.children)
|
||||
validated_sections.append(section)
|
||||
|
||||
return validated_sections
|
||||
|
||||
def _validate_children(self, children: List[Section]) -> List[Section]:
|
||||
"""递归验证子章节"""
|
||||
validated = []
|
||||
for child in children:
|
||||
if self._is_valid_section_with_llm(child):
|
||||
child.children = self._validate_children(child.children)
|
||||
validated.append(child)
|
||||
return validated
|
||||
|
||||
def _is_valid_section_with_llm(self, section: Section) -> bool:
|
||||
"""使用LLM判断章节是否有效"""
|
||||
# 先用规则快速过滤明显无效的章节
|
||||
invalid_titles = [
|
||||
'本文档可作为', '故障', '实时', '输入/输出',
|
||||
'固件', '功能\\', '\\4.', '\\3.'
|
||||
]
|
||||
for invalid in invalid_titles:
|
||||
if invalid in section.title:
|
||||
logger.debug(f"过滤无效章节: {section.number} {section.title}")
|
||||
return False
|
||||
|
||||
# 对于需求相关章节（第3章），额外验证
|
||||
if section.number and section.number.startswith('3'):
|
||||
# 检查标题是否看起来像是有效的需求章节标题
|
||||
# 有效的标题应该是完整的中文短语
|
||||
if '\\' in section.title or '/' in section.title:
|
||||
if not any(kw in section.title for kw in ['输入', '输出', '接口']):
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def create_parser(file_path: str) -> DocumentParser:
|
||||
"""
|
||||
工厂函数：根据文件扩展名创建相应的解析器
|
||||
"""
|
||||
ext = Path(file_path).suffix.lower()
|
||||
|
||||
if ext == '.docx':
|
||||
return DocxParser(file_path)
|
||||
elif ext == '.pdf':
|
||||
return PDFParser(file_path)
|
||||
else:
|
||||
raise ValueError(f"不支持的文件格式: {ext}")
|
||||
198
rag-web-ui/backend/app/tools/srs_reqs_qwen/src/json_generator.py
Normal file
198
rag-web-ui/backend/app/tools/srs_reqs_qwen/src/json_generator.py
Normal file
@@ -0,0 +1,198 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
JSON生成器模块 - LLM增强版
|
||||
将提取的需求和章节结构转换为结构化JSON输出
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Any, Optional
|
||||
from .document_parser import Section
|
||||
from .requirement_extractor import Requirement
|
||||
from .settings import AppSettings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class JSONGenerator:
|
||||
"""JSON输出生成器"""
|
||||
|
||||
def __init__(self, config: Dict = None):
|
||||
self.config = config or {}
|
||||
self.settings = AppSettings(self.config)
|
||||
|
||||
def generate(self, sections: List[Section], requirements: List[Requirement],
|
||||
document_title: str = "SRS Document") -> Dict[str, Any]:
|
||||
"""
|
||||
生成JSON输出
|
||||
|
||||
Args:
|
||||
sections: 章节列表
|
||||
requirements: 需求列表
|
||||
document_title: 文档标题
|
||||
|
||||
Returns:
|
||||
结构化JSON字典
|
||||
"""
|
||||
# 按章节组织需求
|
||||
reqs_by_section = self._group_requirements_by_section(requirements)
|
||||
|
||||
# 统计需求类型
|
||||
type_stats = self._calculate_type_statistics(requirements)
|
||||
|
||||
# 构建输出结构
|
||||
output = {
|
||||
"文档元数据": {
|
||||
"标题": document_title,
|
||||
"生成时间": datetime.now().isoformat(),
|
||||
"总需求数": len(requirements),
|
||||
"需求类型统计": type_stats
|
||||
},
|
||||
"需求内容": self._build_requirement_content(sections, reqs_by_section)
|
||||
}
|
||||
|
||||
logger.info(f"生成JSON输出，共{len(requirements)}个需求")
|
||||
return output
|
||||
|
||||
def _group_requirements_by_section(self, requirements: List[Requirement]) -> Dict[str, List[Requirement]]:
|
||||
"""按章节编号分组需求"""
|
||||
grouped = {}
|
||||
for req in requirements:
|
||||
section_key = req.section_uid or req.section_number or 'unknown'
|
||||
if section_key not in grouped:
|
||||
grouped[section_key] = []
|
||||
grouped[section_key].append(req)
|
||||
return grouped
|
||||
|
||||
def _calculate_type_statistics(self, requirements: List[Requirement]) -> Dict[str, int]:
|
||||
"""计算需求类型统计"""
|
||||
stats = {}
|
||||
for req in requirements:
|
||||
type_chinese = self.settings.type_chinese.get(req.type, '其他需求')
|
||||
if type_chinese not in stats:
|
||||
stats[type_chinese] = 0
|
||||
stats[type_chinese] += 1
|
||||
return stats
|
||||
|
||||
def _should_include_section(self, section: Section) -> bool:
|
||||
"""判断章节是否应该包含在输出中"""
|
||||
return not self.settings.is_non_requirement_section(section.title)
|
||||
|
||||
def _build_requirement_content(self, sections: List[Section],
|
||||
reqs_by_section: Dict[str, List[Requirement]]) -> Dict[str, Any]:
|
||||
"""构建需求内容的层次结构"""
|
||||
content = {}
|
||||
|
||||
for section in sections:
|
||||
# 只处理需求相关章节
|
||||
if not self._should_include_section(section):
|
||||
# 但仍需检查子章节
|
||||
for child in section.children:
|
||||
child_content = self._build_section_content_recursive(child, reqs_by_section)
|
||||
if child_content:
|
||||
key = f"{child.number} {child.title}" if child.number else child.title
|
||||
content[key] = child_content
|
||||
continue
|
||||
|
||||
section_content = self._build_section_content_recursive(section, reqs_by_section)
|
||||
if section_content:
|
||||
key = f"{section.number} {section.title}" if section.number else section.title
|
||||
content[key] = section_content
|
||||
|
||||
return content
|
||||
|
||||
def _build_section_content_recursive(self, section: Section,
|
||||
reqs_by_section: Dict[str, List[Requirement]]) -> Optional[Dict[str, Any]]:
|
||||
"""递归构建章节内容"""
|
||||
# 检查是否应该包含此章节
|
||||
if not self._should_include_section(section):
|
||||
return None
|
||||
|
||||
# 章节基本信息
|
||||
result = {
|
||||
"章节信息": {
|
||||
"章节编号": section.number or "",
|
||||
"章节标题": section.title,
|
||||
"章节级别": section.level
|
||||
}
|
||||
}
|
||||
|
||||
# 检查是否有子章节
|
||||
has_valid_children = False
|
||||
subsections = {}
|
||||
|
||||
for child in section.children:
|
||||
child_content = self._build_section_content_recursive(child, reqs_by_section)
|
||||
if child_content:
|
||||
has_valid_children = True
|
||||
key = f"{child.number} {child.title}" if child.number else child.title
|
||||
subsections[key] = child_content
|
||||
|
||||
# 添加当前章节需求
|
||||
reqs = reqs_by_section.get(section.uid or section.number or 'unknown', [])
|
||||
reqs = sorted(reqs, key=lambda r: getattr(r, 'source_order', 0))
|
||||
if reqs:
|
||||
result["需求列表"] = []
|
||||
for req in reqs:
|
||||
# 需求类型放在最前面
|
||||
type_chinese = self.settings.type_chinese.get(req.type, '功能需求')
|
||||
req_dict = {
|
||||
"需求类型": type_chinese,
|
||||
"需求编号": req.id,
|
||||
"需求描述": req.description
|
||||
}
|
||||
# 接口需求增加额外字段
|
||||
if req.type == 'interface':
|
||||
req_dict["接口名称"] = req.interface_name
|
||||
req_dict["接口类型"] = req.interface_type
|
||||
req_dict["来源"] = req.source
|
||||
req_dict["目的地"] = req.destination
|
||||
result["需求列表"].append(req_dict)
|
||||
|
||||
# 如果有子章节，添加子章节
|
||||
if has_valid_children:
|
||||
result["子章节"] = subsections
|
||||
|
||||
# 如果章节既没有需求也没有子章节，返回None
|
||||
if "需求列表" not in result and "子章节" not in result:
|
||||
return None
|
||||
|
||||
return result
|
||||
|
||||
def save_to_file(self, output: Dict[str, Any], file_path: str) -> None:
|
||||
"""
|
||||
将输出保存到文件
|
||||
|
||||
Args:
|
||||
output: 输出字典
|
||||
file_path: 输出文件路径
|
||||
"""
|
||||
try:
|
||||
output_cfg = self.config.get("output", {})
|
||||
indent = output_cfg.get("indent", 2)
|
||||
pretty = output_cfg.get("pretty_print", True)
|
||||
with open(file_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(output, f, ensure_ascii=False, indent=indent if pretty else None)
|
||||
logger.info(f"成功保存JSON到: {file_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"保存JSON文件失败: {e}")
|
||||
raise
|
||||
|
||||
def generate_and_save(self, sections: List[Section], requirements: List[Requirement],
|
||||
document_title: str, file_path: str) -> Dict[str, Any]:
|
||||
"""
|
||||
生成并保存JSON
|
||||
|
||||
Args:
|
||||
sections: 章节列表
|
||||
requirements: 需求列表
|
||||
document_title: 文档标题
|
||||
file_path: 输出文件路径
|
||||
|
||||
Returns:
|
||||
生成的输出字典
|
||||
"""
|
||||
output = self.generate(sections, requirements, document_title)
|
||||
self.save_to_file(output, file_path)
|
||||
return output
|
||||
197
rag-web-ui/backend/app/tools/srs_reqs_qwen/src/llm_interface.py
Normal file
197
rag-web-ui/backend/app/tools/srs_reqs_qwen/src/llm_interface.py
Normal file
@@ -0,0 +1,197 @@
|
||||
# src/llm_interface.py
|
||||
"""
|
||||
LLM接口模块 - 支持多个LLM提供商
|
||||
"""
|
||||
|
||||
import logging
|
||||
import json
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
from .utils import get_env_or_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMInterface(ABC):
|
||||
"""LLM接口基类"""
|
||||
|
||||
def __init__(self, api_key: str = None, model: str = None, **kwargs):
|
||||
"""
|
||||
初始化LLM接口
|
||||
|
||||
Args:
|
||||
api_key: API密钥
|
||||
model: 模型名称
|
||||
**kwargs: 其他参数（如temperature, max_tokens等）
|
||||
"""
|
||||
self.api_key = api_key
|
||||
self.model = model
|
||||
self.extra_params = kwargs
|
||||
|
||||
@abstractmethod
|
||||
def call(self, prompt: str) -> str:
|
||||
"""
|
||||
调用LLM API
|
||||
|
||||
Args:
|
||||
prompt: 提示词
|
||||
|
||||
Returns:
|
||||
LLM的响应文本
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def call_json(self, prompt: str) -> Dict[str, Any]:
|
||||
"""
|
||||
调用LLM API并获取JSON格式的响应
|
||||
|
||||
Args:
|
||||
prompt: 提示词
|
||||
|
||||
Returns:
|
||||
解析后的JSON字典
|
||||
"""
|
||||
pass
|
||||
|
||||
def validate_config(self) -> bool:
|
||||
"""验证配置是否完整"""
|
||||
return bool(self.api_key and self.model)
|
||||
|
||||
|
||||
class QwenLLM(LLMInterface):
|
||||
"""阿里云千问LLM实现"""
|
||||
|
||||
def __init__(self, api_key: str = None, model: str = "qwen-plus",
|
||||
api_endpoint: str = None, **kwargs):
|
||||
"""
|
||||
初始化千问LLM
|
||||
|
||||
Args:
|
||||
api_key: 阿里云API密钥
|
||||
model: 模型名称（如qwen-plus, qwen-turbo）
|
||||
api_endpoint: API端点地址
|
||||
**kwargs: 其他参数
|
||||
"""
|
||||
super().__init__(api_key, model, **kwargs)
|
||||
self.api_endpoint = api_endpoint or "https://dashscope.aliyuncs.com/compatible-mode/v1"
|
||||
self._check_dashscope_import()
|
||||
|
||||
def _check_dashscope_import(self) -> None:
|
||||
"""检查dashscope库是否已安装"""
|
||||
try:
|
||||
import dashscope
|
||||
self.dashscope = dashscope
|
||||
except ImportError:
|
||||
logger.error("dashscope库未安装，请运行: pip install dashscope")
|
||||
raise
|
||||
|
||||
def call(self, prompt: str) -> str:
|
||||
"""
|
||||
调用千问LLM
|
||||
|
||||
Args:
|
||||
prompt: 提示词
|
||||
|
||||
Returns:
|
||||
LLM的响应文本
|
||||
"""
|
||||
if not self.validate_config():
|
||||
raise ValueError("LLM配置不完整（api_key或model未设置）")
|
||||
|
||||
try:
|
||||
from dashscope import Generation
|
||||
|
||||
# 设置API密钥
|
||||
self.dashscope.api_key = self.api_key
|
||||
|
||||
# 构建请求参数 - dashscope 1.7.0 格式
|
||||
response = Generation.call(
|
||||
model=self.model,
|
||||
messages=[
|
||||
{'role': 'user', 'content': prompt}
|
||||
],
|
||||
result_format='message' # 使用message格式
|
||||
)
|
||||
|
||||
# 调试输出
|
||||
logger.debug(f"API响应类型: {type(response)}")
|
||||
logger.debug(f"API响应内容: {response}")
|
||||
|
||||
# 处理响应
|
||||
if isinstance(response, dict):
|
||||
# dict格式响应
|
||||
status_code = response.get('status_code', 200)
|
||||
if status_code == 200:
|
||||
output = response.get('output', {})
|
||||
if 'choices' in output:
|
||||
return output['choices'][0]['message']['content']
|
||||
elif 'text' in output:
|
||||
return output['text']
|
||||
else:
|
||||
# 尝试直接获取text
|
||||
return output.get('text', str(output))
|
||||
else:
|
||||
error_msg = response.get('message', response.get('code', 'Unknown error'))
|
||||
logger.error(f"千问API返回错误: {error_msg}")
|
||||
raise Exception(f"API调用失败: {error_msg}")
|
||||
else:
|
||||
# 对象格式响应
|
||||
if hasattr(response, 'status_code') and response.status_code == 200:
|
||||
output = response.output
|
||||
if hasattr(output, 'choices'):
|
||||
return output.choices[0].message.content
|
||||
elif hasattr(output, 'text'):
|
||||
return output.text
|
||||
else:
|
||||
return str(output)
|
||||
elif hasattr(response, 'status_code'):
|
||||
error_msg = getattr(response, 'message', str(response))
|
||||
raise Exception(f"API调用失败: {error_msg}")
|
||||
else:
|
||||
return str(response)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"调用千问LLM失败: {e}")
|
||||
raise
|
||||
|
||||
def call_json(self, prompt: str) -> Dict[str, Any]:
|
||||
"""
|
||||
调用千问LLM并获取JSON格式响应
|
||||
|
||||
Args:
|
||||
prompt: 提示词
|
||||
|
||||
Returns:
|
||||
解析后的JSON字典
|
||||
"""
|
||||
# 添加JSON格式要求到提示词
|
||||
json_prompt = prompt + "\n\n请确保响应是有效的JSON格式。"
|
||||
|
||||
response = self.call(json_prompt)
|
||||
|
||||
try:
|
||||
# 尝试解析JSON
|
||||
# 首先尝试直接解析
|
||||
return json.loads(response)
|
||||
except json.JSONDecodeError:
|
||||
# 尝试提取JSON代码块
|
||||
try:
|
||||
import re
|
||||
# 查找JSON代码块
|
||||
json_match = re.search(r'```json\s*(.*?)\s*```', response, re.DOTALL)
|
||||
if json_match:
|
||||
return json.loads(json_match.group(1))
|
||||
|
||||
# 尝试查找任何JSON对象
|
||||
json_match = re.search(r'\{.*\}', response, re.DOTALL)
|
||||
if json_match:
|
||||
return json.loads(json_match.group(0))
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"无法从响应中提取JSON: {e}")
|
||||
|
||||
# 如果都失败，返回错误信息
|
||||
logger.error(f"无法解析LLM响应为JSON: {response}")
|
||||
return {"error": "Failed to parse response as JSON", "raw_response": response}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,74 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
需求编号生成与提取工具。
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import Optional, Tuple, Dict
|
||||
|
||||
|
||||
class RequirementIDGenerator:
|
||||
def __init__(self, type_prefix: Dict[str, str]):
|
||||
self.type_prefix = type_prefix
|
||||
|
||||
def normalize(self, req_id: str) -> str:
|
||||
if not req_id:
|
||||
return ""
|
||||
return str(req_id).strip()
|
||||
|
||||
def extract_from_text(self, text: str) -> Tuple[Optional[str], str]:
|
||||
if not text:
|
||||
return None, text
|
||||
|
||||
pattern1 = r"^\s*([A-Za-z]{2,10}[-_]\d+(?:[-.\d]+)*)\s*[:：\)\]】]?\s*(.+)$"
|
||||
match = re.match(pattern1, text)
|
||||
if match:
|
||||
return match.group(1).strip(), match.group(2).strip()
|
||||
|
||||
pattern2 = r"^\s*([A-Za-z]\d+)\s*[:：\)\]】]?\s*(.+)$"
|
||||
match = re.match(pattern2, text)
|
||||
if match:
|
||||
return match.group(1).strip(), match.group(2).strip()
|
||||
|
||||
pattern3 = r"^\s*([a-z0-9]{1,2}[\)）])\s*(.+)$"
|
||||
match = re.match(pattern3, text)
|
||||
if match:
|
||||
code = match.group(1).strip().rstrip(")）")
|
||||
return code, match.group(2).strip()
|
||||
|
||||
return None, text
|
||||
|
||||
def generate(
|
||||
self,
|
||||
req_type: str,
|
||||
section_number: str,
|
||||
index: int,
|
||||
doc_req_id: str = "",
|
||||
parent_req_id: str = "",
|
||||
split_index: int = 1,
|
||||
split_total: int = 1,
|
||||
) -> str:
|
||||
base_id = self._generate_base(req_type, section_number, index, doc_req_id, parent_req_id)
|
||||
if split_total > 1:
|
||||
return f"{base_id}-S{split_index}"
|
||||
return base_id
|
||||
|
||||
def _generate_base(
|
||||
self,
|
||||
req_type: str,
|
||||
section_number: str,
|
||||
index: int,
|
||||
doc_req_id: str,
|
||||
parent_req_id: str,
|
||||
) -> str:
|
||||
if doc_req_id:
|
||||
complete_id_pattern = r"^[A-Za-z0-9]{2,10}[-_].+$"
|
||||
if re.match(complete_id_pattern, doc_req_id):
|
||||
return doc_req_id.replace("_", "-")
|
||||
|
||||
if doc_req_id and parent_req_id:
|
||||
return f"{parent_req_id}-{doc_req_id}"
|
||||
|
||||
prefix = self.type_prefix.get(req_type, "FR")
|
||||
section_part = section_number if section_number else "NA"
|
||||
return f"{prefix}-{section_part}-{index}"
|
||||
@@ -0,0 +1,188 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
需求长句拆分器。
|
||||
将复合长句拆分为可验证的原子需求片段。
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import List
|
||||
|
||||
|
||||
class RequirementSplitter:
|
||||
ACTION_HINTS = [
|
||||
"产生",
|
||||
"发送",
|
||||
"设置",
|
||||
"进入",
|
||||
"退出",
|
||||
"关闭",
|
||||
"开启",
|
||||
"监测",
|
||||
"判断",
|
||||
"记录",
|
||||
"上传",
|
||||
"重启",
|
||||
"恢复",
|
||||
"关断",
|
||||
"断电",
|
||||
"加电",
|
||||
"执行",
|
||||
"进行",
|
||||
]
|
||||
|
||||
CONNECTOR_HINTS = ["并", "并且", "同时", "然后", "且", "以及", "及"]
|
||||
CONDITIONAL_HINTS = ["如果", "当", "若", "在", "其中", "此时", "满足"]
|
||||
CONTEXT_PRONOUN_HINTS = ["该", "其", "上述", "此", "这些", "那些"]
|
||||
|
||||
def __init__(self, max_sentence_len: int = 120, min_clause_len: int = 12):
|
||||
self.max_sentence_len = max_sentence_len
|
||||
self.min_clause_len = min_clause_len
|
||||
|
||||
def split(self, text: str) -> List[str]:
|
||||
cleaned = self._clean(text)
|
||||
if not cleaned:
|
||||
return []
|
||||
|
||||
if self._contains_strong_semantic_chain(cleaned):
|
||||
return [cleaned]
|
||||
|
||||
# 先按强分隔符切分为主片段。
|
||||
base_parts = self._split_by_strong_punctuation(cleaned)
|
||||
|
||||
result: List[str] = []
|
||||
for part in base_parts:
|
||||
if len(part) <= self.max_sentence_len:
|
||||
result.append(part)
|
||||
continue
|
||||
|
||||
# 对超长片段进一步基于逗号和连接词拆分。
|
||||
refined = self._split_long_clause(part)
|
||||
result.extend(refined)
|
||||
|
||||
result = self._merge_semantic_chain(result)
|
||||
result = self._merge_too_short(result)
|
||||
return self._deduplicate(result)
|
||||
|
||||
def _contains_strong_semantic_chain(self, text: str) -> bool:
|
||||
# 条件-动作链完整时，避免强拆。
|
||||
has_conditional = any(h in text for h in ["如果", "若", "当"])
|
||||
has_result = "则" in text or "时" in text
|
||||
action_count = sum(1 for h in self.ACTION_HINTS if h in text)
|
||||
if has_conditional and has_result and action_count >= 2:
|
||||
return True
|
||||
return False
|
||||
|
||||
def _clean(self, text: str) -> str:
|
||||
text = re.sub(r"\s+", " ", text or "")
|
||||
return text.strip(" ;；。")
|
||||
|
||||
def _split_by_strong_punctuation(self, text: str) -> List[str]:
|
||||
chunks = re.split(r"[；;。]", text)
|
||||
return [c.strip(" ，,") for c in chunks if c and c.strip(" ，,")]
|
||||
|
||||
def _split_long_clause(self, clause: str) -> List[str]:
|
||||
if self._contains_strong_semantic_chain(clause):
|
||||
return [clause]
|
||||
|
||||
raw_parts = [x.strip() for x in re.split(r"[，,]", clause) if x.strip()]
|
||||
if len(raw_parts) <= 1:
|
||||
return [clause]
|
||||
|
||||
assembled: List[str] = []
|
||||
current = raw_parts[0]
|
||||
|
||||
for fragment in raw_parts[1:]:
|
||||
if self._should_split(current, fragment):
|
||||
assembled.append(current.strip())
|
||||
current = fragment
|
||||
else:
|
||||
current = f"{current}，{fragment}"
|
||||
|
||||
if current.strip():
|
||||
assembled.append(current.strip())
|
||||
|
||||
return assembled
|
||||
|
||||
def _should_split(self, current: str, fragment: str) -> bool:
|
||||
if len(current) < self.min_clause_len:
|
||||
return False
|
||||
|
||||
# 指代承接片段通常是语义延续，不应切断。
|
||||
if any(fragment.startswith(h) for h in self.CONTEXT_PRONOUN_HINTS):
|
||||
return False
|
||||
|
||||
# 条件链中带“则/并/同时”的后继片段，优先保持在同一需求中。
|
||||
if self._contains_strong_semantic_chain(current + "，" + fragment):
|
||||
return False
|
||||
|
||||
frag_starts_with_condition = any(fragment.startswith(h) for h in self.CONDITIONAL_HINTS)
|
||||
if frag_starts_with_condition:
|
||||
return False
|
||||
|
||||
has_connector = any(fragment.startswith(h) for h in self.CONNECTOR_HINTS)
|
||||
has_action = any(h in fragment for h in self.ACTION_HINTS)
|
||||
current_has_action = any(h in current for h in self.ACTION_HINTS)
|
||||
|
||||
# 连接词 + 动作词，且当前片段已经包含动作，优先拆分。
|
||||
if has_connector and has_action and current_has_action:
|
||||
return True
|
||||
|
||||
# 无连接词但出现新的动作片段且整体过长，也拆分。
|
||||
if has_action and current_has_action and len(current) >= self.max_sentence_len // 2:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _merge_semantic_chain(self, parts: List[str]) -> List[str]:
|
||||
if not parts:
|
||||
return []
|
||||
|
||||
merged: List[str] = [parts[0]]
|
||||
for part in parts[1:]:
|
||||
prev = merged[-1]
|
||||
if self._should_merge(prev, part):
|
||||
merged[-1] = f"{prev}；{part}"
|
||||
else:
|
||||
merged.append(part)
|
||||
return merged
|
||||
|
||||
def _should_merge(self, prev: str, current: str) -> bool:
|
||||
# 指代开头：如“该报警信号...”。
|
||||
if any(current.startswith(h) for h in self.CONTEXT_PRONOUN_HINTS):
|
||||
return True
|
||||
|
||||
# 报警触发后的持续条件与动作属于同一链。
|
||||
if ("报警" in prev and "持续" in current) or ("产生" in prev and "报警" in prev and "持续" in current):
|
||||
return True
|
||||
|
||||
# 状态迁移 + 后续控制动作保持合并。
|
||||
if ("进入" in prev or "设置" in prev or "发送" in prev) and ("则" in current or "连续" in current):
|
||||
return True
|
||||
|
||||
# 条件链分裂片段重新合并。
|
||||
if self._contains_strong_semantic_chain(prev + "，" + current):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _merge_too_short(self, parts: List[str]) -> List[str]:
|
||||
if not parts:
|
||||
return []
|
||||
|
||||
merged: List[str] = []
|
||||
for part in parts:
|
||||
if merged and len(part) < self.min_clause_len:
|
||||
merged[-1] = f"{merged[-1]}，{part}"
|
||||
else:
|
||||
merged.append(part)
|
||||
return merged
|
||||
|
||||
def _deduplicate(self, parts: List[str]) -> List[str]:
|
||||
seen = set()
|
||||
result = []
|
||||
for part in parts:
|
||||
key = re.sub(r"\s+", "", part)
|
||||
if key and key not in seen:
|
||||
seen.add(key)
|
||||
result.append(part)
|
||||
return result
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user