Update LLM API Router Service
1 parent
d949f1153a
commit
a73e41e7cd
1 file changed
+2
-2
@@ -32,7 +32,7 @@ configuration is provided through a `LLMApiRouterConfiguration` object.
|
||||
`AllLLMServicesUnavailableException`.
|
||||
* The request is routed to the selected `LLMService`.
|
||||
* This call is wrapped in the `autoRetry` method
|
||||
* depending on the endpoint, the `shortTimeout` parameter is set
|
||||
* The `shortTimeout` parameter is set based on the endpoint.
|
||||
* `true` for `processQuery`
|
||||
* `false` for `generateSummary`
|
||||
|
||||
@@ -49,7 +49,7 @@ configuration is provided through a `LLMApiRouterConfiguration` object.
|
||||
|
||||
* If a timeout is encountered by an `LLMService`, it sets its `operational` flag to false and throws a `ServiceUnavailableException`
|
||||
* the autoRetry method catches the exception and automatically retries the request
|
||||
* the `LLMApiRouter` always selects the first available service for each attempt independently each attempt independently
|
||||
* the `LLMApiRouter` always selects the first available service for each attempt independently
|
||||
|
||||
5. **Service Health Checks:**
|
||||
* A scheduled task (`@Scheduled`) periodically checks the availability of each `LLMService`.
|
||||
|
||||
Reference in new issue
Block a user