chatty-coder logo

The chatty-coder is an agent-based human-in-the-loop computer program code generator. It is based on the idea introduced in the 2024 paper Can Language Models Solve Olympiad Programming?1, which uses a combination of self-reflection and retrieval over episodic knowledge with human-in-the-loop collaboration to improve code generation responses of language models to coding challenges from the USA Computing Olympiad (USACO). The chatty-coder resembles the functionality of the experimental LangGraph-based implementation, called Competitive Programming, of the aforementioned paper. The chatty-coder project has a Gradio-based user interface and an associated FastAPI-based REST API.

1

Shi, Q., Tang, M., Narasimhan, K. and Yao, S., 2024. Can Language Models Solve Olympiad Programming? arXiv preprint arXiv:2404.10952.

Python dependencies

You will need Python installed on your computer. The code in this repository has been tested on Python 3.12.0. Refrain from using the system Python. Use a Python version and virtual environment manager such as pyenv. Create a new Python virtual environment for Python 3.12.0 or above. For example, this is how you do it using pyenv. The second line activates the virtual environment.

# Uncomment the following line to install Python 3.12.0
# pyenv install 3.12.0
pyenv virtualenv 3.12.0 chatty-coder
pyenv activate chatty-coder

Once the virtual environment is setup, you can install all the dependencies for this application in it by running the following.

pip install -U pip
pip install -U -r requirements.txt

Optional: Uninstall all dependencies to start afresh

If necessary, you can uninstall everything previously installed by pip (in a virtual environment) by running the following.

pip freeze | xargs pip uninstall -y

Container (Docker)

You can run the app in a Docker container. To do so, you have to build its image (which we name as chatty-coder although you can choose any other name) and run an instance (which we name as chatty-coder-container but you can also pick a name of your choice) of that image, as follows.

docker build -f local.dockerfile -t chatty-coder .
docker create -p 7860:7860/tcp --name chatty-coder-container chatty-coder

Configuration through environment variables

The app configuration can be controlled using environment variables. You can specify those variables using a .env file, for example. If you are containerising the app then you must update the .env.docker before creating the Docker image.

The following is a list of configuration variables. All environment variables must be specified as quoted strings, such as ENV_KEY = "value". Type casting will be performed by the app as necessary. Some of the variables are required or conditionally required.

Gradio server settings

Notice that host and port variables listed below are not the same as the corresponding ones mentioned in Gradio documentation, which will be ignored.

  • GRADIO__SERVER_HOST: The host name to which the Gradio server should bind to. Default value: localhost. Set to 0.0.0.0 in .env.docker.
  • GRADIO__SERVER_PORT: The port number on the GRADIO__SERVER_HOST, which the Gradio server should bind on. Default value: 7860. Parsed as non-string type: int.

General large language model (LLM) settings

  • LLM__PROVIDER: The provider for the language language models. Default value: Ollama. Possible values are: Open AI, Cohere, Groq and Ollama.
  • LLM__TEMPERATURE: The temperature parameter for the LLM to be used. The value is usually a decimal number between 0 and 1 but the actual range depends on the model. For example, for Open AI models, the value can be between 0 and 2. Default value: 0.4. Parsed as non-string type: float.
  • LLM__TOP_P: The top_p paraeter for the LLM to be used, between 0 and 1. Default value: 0.4. Parsed as non-string type: float.
  • LLM__TOP_K: The top_k parameter for the LLM to be used. Default value: 40. Parsed as non-string type: int.
  • LLM__REPEAT_PENALTY: The repeat penalty parameter for the LLM to be used. Default value: 1.1. Parsed as non-string type: float.
  • LLM__SEED: The seed parameter for the LLM to be used. Default value: 1. Parsed as non-string type: int.

For details about these parameter settings, see the documentation of your chosen LLM provider, such as, the Ollama documentation for the valid parameters and their values for model files. Note that for language model providers other than Ollama, all model parameter settings apart from LLM__TEMPERATURE are ignored.

Language model specific settings

  • LLM__OPENAI_API_KEY: (required only if language model provider is Open AI) A valid Open AI API key.
  • LLM__OPENAI_MODEL: The Open AI model you want to use. Default value: gpt-4o-mini.
  • LLM__COHERE_API_KEY: (required only if language model provider is Cohere) A valid Cohere API key.
  • LLM__COHERE_MODEL: The Cohere model you want to use. Default value: command-r-plus.
  • LLM__GROQ_API_KEY: (required only if language model provider is Groq) A valid Groq API key.
  • LLM__GROQ_MODEL: The Groq model you want to use. Default value: llama3-groq-70b-8192-tool-use-preview.
  • LLM__OLLAMA_URL: URL for a running Ollama instance. Default value: http://localhost:11434. Set to http://host.docker.internal:11434 in .env.docker. (Learn about Docker networking and set it correctly as needed.)
  • LLM__OLLAMA_MODEL: The Ollama model you want to use. Default value: llama3.

Run locally

The non-containerised way

Once you have installed the dependencies mentioned above in your Python virtual environment, to run the chatbot, execute python src/app.py. If the web server starts up without any problem, the web application user interface will be available at the host and port specified by the environment variables GRADIO__SERVER_HOST and GRADIO__SERVER_PORT respectively, assuming that the server can bind to the specified port on the specified host. If you do not specify these variables, they will default to localhost and 7860 respectively, which means your web application will be available at http://localhost:7860.

Containerised (Docker)

Following the creation of the container, you can run the app using docker container start chatty-coder-container the app will be accessible on your Docker host, for example as http://localhost:7860 -- assuming that nothing else on host is blocking port 7860 when the container starts.

The Docker container has to depend on external LLM provider, graph database, document and index storage. If any of these, such as Ollama, is running on the Docker host then you should use the host name for the service as host.docker.internal or gateway.docker.internal. See the networking documentation of Docker desktop for details.