The only problem I really have, is context size. It’s harder to get larger than 8k context size and maintain decent generation speed with 16 GB of VRAM and 16 GB of RAM. Gonna get more RAM at some point though, and hope ollama/llamacpp gets better at memory management. Hopefully the distributed running from llamaccp ends up in ollama.
Is it possible to use ollama or an arbitrary OpenAI-compatible endpoint with the chatbot feature yet? Or only the cloud providers?