Viewing a single comment thread. View all comments

onequark t1_jdx5o96 wrote

Awesome, does it support token streaming?

3

lhenault OP t1_jdxpuso wrote

Not for now but that’s indeed a cool feature and something available in OpenAI API. It shouldn’t be too hard to implement, as I’ve already started something for that on the gRPC backend, and as FastAPI has a StreamingResponse. Thanks for suggesting it, will try to prioritise this!

2