7fbfbb0dc5
Add token streaming using ServerSideEvents (SSE).
The signature of the SSE events is:
```rust
struct Details {
finish_reason: String,
generated_tokens: u32,
seed: Option<u64>,
}
struct StreamResponse {
token: Token,
generated_text: Option<String>,
details: Option<Details>,
}
struct ErrorResponse {
error: String,
}
```
|
||
|---|---|---|
| .. | ||
| tests | ||
| text_generation | ||
| .gitignore | ||
| Makefile | ||
| README.md | ||
| poetry.lock | ||
| pyproject.toml | ||
README.md
BLOOM Inference Python gRPC Server
A Python gRPC server for BLOOM Inference
Install
make install
Run
make run-dev