iCode/docs
1
0
Wiki

Update LLM API Router Service

midnight
2025-09-26 04:09:41 +02:00
parent d949f1153a
commit a73e41e7cd
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
LLM-API-Router-Service.md

@@ -32,7 +32,7 @@ configuration is provided through a `LLMApiRouterConfiguration` object.
`AllLLMServicesUnavailableException`. `AllLLMServicesUnavailableException`.
* The request is routed to the selected `LLMService`. * The request is routed to the selected `LLMService`.
* This call is wrapped in the `autoRetry` method * This call is wrapped in the `autoRetry` method
* depending on the endpoint, the `shortTimeout` parameter is set * The `shortTimeout` parameter is set based on the endpoint.
* `true` for `processQuery` * `true` for `processQuery`
* `false` for `generateSummary` * `false` for `generateSummary`
@@ -49,7 +49,7 @@ configuration is provided through a `LLMApiRouterConfiguration` object.
* If a timeout is encountered by an `LLMService`, it sets its `operational` flag to false and throws a `ServiceUnavailableException` * If a timeout is encountered by an `LLMService`, it sets its `operational` flag to false and throws a `ServiceUnavailableException`
* the autoRetry method catches the exception and automatically retries the request * the autoRetry method catches the exception and automatically retries the request
* the `LLMApiRouter` always selects the first available service for each attempt independently each attempt independently * the `LLMApiRouter` always selects the first available service for each attempt independently
5. **Service Health Checks:** 5. **Service Health Checks:**
* A scheduled task (`@Scheduled`) periodically checks the availability of each `LLMService`. * A scheduled task (`@Scheduled`) periodically checks the availability of each `LLMService`.