drbh
|
cef0553d59
|
Outlines guided generation (#1539)
This WIP PR starts to add grammar support via outlines, currently this
PR supports very simple regex grammars and does not optimize for
precompiling or caching grammar fsm's.
todo:
- [X] add simple outlines guidance to `NextTokenChooser`
- [X] update protos for grammar
- [X] update generation params API
- [X] constrain simple grammar
- [ ] support parsing more complex grammar into fsm
- [ ] support all outline support grammar types
- [ ] explore optimizations to avoid recompiling grammars
guided request
```bash
curl -s 'http://localhost:3000/generate' \
--header 'Content-Type: application/json' \
--data-raw '{
"inputs": "make an email for david: \n",
"parameters": {
"max_new_tokens": 6,
"grammar": "[\\w-]+@([\\w-]+\\.)+[\\w-]+"
}
}' | jq
```
response
```json
{
"generated_text": "david@example.com"
}
```
unguided request
```bash
curl -s 'http://localhost:3000/generate' \
--header 'Content-Type: application/json' \
--data '{
"inputs": "make an email for david: \n",
"parameters": {
"max_new_tokens": 6
}
}' | jq
```
response
```json
{
"generated_text": " email = 'david"
}
```
|
2024-02-15 10:28:10 +01:00 |
|
OlivierDehaene
|
0d794af6a5
|
feat: experimental support for cuda graphs (#1428)
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
|
2024-02-12 10:09:29 +01:00 |
|
OlivierDehaene
|
44b267ab22
|
fix: fix gpt-q params loading
|
2023-12-14 11:02:16 +01:00 |
|
OlivierDehaene
|
82670d9786
|
feat: add quant to mixtral (#1337)
|
2023-12-12 17:55:03 +01:00 |
|
OlivierDehaene
|
72ee382ded
|
chore: formatting
|
2023-12-11 14:49:52 +01:00 |
|
OlivierDehaene
|
3a521c92b3
|
feat: mixtral (#1328)
|
2023-12-11 14:43:40 +01:00 |
|
Nicolas Patry
|
9ecfa16b12
|
Speculative (#1308)
|
2023-12-11 12:46:30 +01:00 |
|
OlivierDehaene
|
3b56d7669b
|
feat: add mistral model (#1071)
|
2023-09-28 09:55:47 +02:00 |