Devlog 2: Streaming Responses

Hello everybody! I’m back very quickly with the second devlog. As promised, Oh! Chat is:

IN THE MODERN DIGITAL AGE, it has become increasingly difficult to control one’s exposure to the internet, regardless of how reserved one may be or what precautions one may take. With the advent of the large language model era, where people could discuss confidential information with chatbots, this problem - our inability to control the flow of our own data, relying on vague promises of ‘privacy’ that cloud-based services claim but never prove - has become more relevant than ever. Oh! Chat aims to combat this by giving users the premium UX that one experiences on existing AI chatbot applications and interfaces, retaining as much AI functionality as possible, while facilitating user operation, installation, and guaranteeing data privacy. (from a readme draft)

Anyhow, let’s get to the point. Since we want everyday users to feel comfortable, we want to replicate the experience of existing AI chatbots. An important part of this is the text streaming feature, which allows users to see the response as it is being generated: a typing-like effect, and it is also a good method to verify that the model is working. This is what I did in this devlog.

What did I do?

I:

1. Implemented a streaming response feature in the backend.
   • Firstly, we ditch the original old logic with match request.await. This is because the await means we have to WAIT for the entire response to output before continued function execution (which is unideal).
   • Instead, we use reqwest::Client to send a POST request to the Ollama instance, and then we use response.bytes_stream() to get a stream of bytes from the response. We then iterate over the stream using while let Some(chunk) = stream.next().await, and each chunk we convert it to a string and send it back to the frontend using emit_all("chunk", chunk_str). And the client gets it in real time via the listen("chunk", (event) => { ... }) function as an initialised await in main.js. Finally, at the end of the response, a rust event response_end is emitted, and the frontend can accordingly adjust. A very simple implementation, but it works effectively.
2. Drafted a basic readme.
3. … That’s it. (It was quite hard to implement, but it is necessary for user experience.)

What’s next?

I plan on:

1. Allowing chat customisation (to some degree) via settings. More details to come later.
2. UI overhaul no. 1 and replacing the boilerplate html structure with a more well-defined one.

Anyhow, thanks for reading! Watch for the next devlog.