Create an environment for prompt-based methods (Python 3.10 is **required**)
Create an environment for prompt-based methods (Python 3.10 is **required**)
```shell
```shell
@ -48,6 +49,7 @@ To deactivate the conda environment, run: (Only after running all the experiment
```shell
```shell
conda deactivate
conda deactivate
```
```
#### Download and extract SOLOIST pre-trained model
#### Download and extract SOLOIST pre-trained model
Download and unzip the pretrained model, this is used for fine-tuning the baseline and prompt-based methods. For more details about the pre-trained SOLOIST model, refer to the GitHub [repo](https://github.com/pengbaolin/soloist).
Download and unzip the pretrained model, this is used for fine-tuning the baseline and prompt-based methods. For more details about the pre-trained SOLOIST model, refer to the GitHub [repo](https://github.com/pengbaolin/soloist).
@ -66,10 +68,12 @@ Clone the repository for source code
Pull the changes from remote (if local is behind the remote)
Pull the changes from remote (if local is behind the remote)
```shell
```shell
git pull
git pull
```
```
Change directory
Change directory
```shell
```shell
cd master-thesis
cd master-thesis
@ -120,7 +124,8 @@ pip install requirements.txt
Train a separate model for each data split. Edit the [train_baseline.sh](baseline/train_baseline.sh) file to modify the hyperparameters while training (learning rate, epochs). Use `CUDA_VISIBLE_DEVICES` to specify a CUDA device (GPU) for training the model.
Train a separate model for each data split. Edit the [train_baseline.sh](baseline/train_baseline.sh) file to modify the hyperparameters while training (learning rate, epochs). Use `CUDA_VISIBLE_DEVICES` to specify a CUDA device (GPU) for training the model.
```shell
```shell
sh train_baseline.sh -d <data-split-name>
sh train_baseline.sh -d <data-split-name>
```
```
Pass the data split name to `-d` flag. Possible values are: `5-dpd`, `10-dpd`, `50-dpd`, `100-dpd`, `125-dpd`, `250-dpd`
Pass the data split name to `-d` flag. Possible values are: `5-dpd`, `10-dpd`, `50-dpd`, `100-dpd`, `125-dpd`, `250-dpd`
Example training command: `sh train_baseline.sh -d 50-dpd`
Example training command: `sh train_baseline.sh -d 50-dpd`
@ -131,13 +136,15 @@ Choose a checkpoint of the saved baseline model to generate belief state predict
Set the `MODEL_CHECKPOINT` environment variable with the path to the chosen model checkpoint. It should only contain the path from the "experiment-{datetime}" folder.
Set the `MODEL_CHECKPOINT` environment variable with the path to the chosen model checkpoint. It should only contain the path from the "experiment-{datetime}" folder.
The generated predictions are saved under `OUTPUTS_DIR_BASELINE` folder. Some of the generated belief state predictions are uploaded to this repository and can be found under [outputs](outputs) folder.
The generated predictions are saved under `OUTPUTS_DIR_BASELINE` folder. Some of the generated belief state predictions are uploaded to this repository and can be found under [outputs](outputs) folder.
### Baseline Evaluation
### Baseline Evaluation
@ -147,7 +154,8 @@ The standard Joint Goal Accuracy (JGA) is used to evaluate the belief prediction
Edit the [evaluate.py](baseline/evaluate.py) to set the predictions output file before running the evaluation
Edit the [evaluate.py](baseline/evaluate.py) to set the predictions output file before running the evaluation
```shell
```shell
python evaluate.py
python evaluate.py
```
```
### Results from baseline experiments
### Results from baseline experiments
|data-split| JGA |
|data-split| JGA |
|--|:--:|
|--|:--:|
@ -171,21 +179,27 @@ Change directory to `prompt-learning` and install the requirements. Make sure th
```shell
```shell
cd prompt-learning
cd prompt-learning
pip install requirements.txt
pip install requirements.txt
```
```
### Train the prompt model
### Train the prompt model
Train a separate model for each data split. Edit the [train_prompting.sh](prompt-learning/train_prompting.sh) file to modify the default hyperparameters for training (learning rate, epochs).
Train a separate model for each data split. Edit the [train_prompting.sh](prompt-learning/train_prompting.sh) file to modify the default hyperparameters for training (learning rate, epochs).
```shell
```shell
sh train_prompting.sh -d <data-split-name>
sh train_prompting.sh -d <data-split-name>
```
```
Pass the data split name to `-d` flag.
Pass the data split name to `-d` flag.
Possible values are: `5-dpd`, `10-dpd`, `50-dpd`, `100-dpd`, `125-dpd`, `250-dpd`
Possible values are: `5-dpd`, `10-dpd`, `50-dpd`, `100-dpd`, `125-dpd`, `250-dpd`
Example training command: `sh train_baseline.sh -d 50-dpd`
Example training command: `sh train_baseline.sh -d 50-dpd`
**Some `train_prompting.sh` flags**:
**Some `train_prompting.sh` flags**:
`--num_epochs 10` - Number of epochs
`--num_epochs 10` - Number of epochs
`--learning_rate 5e-5` - Initial learning rate for Optimizer
`--learning_rate 5e-5` - Initial learning rate for Optimizer
`--with_inverse_prompt` - Use Inverse Prompt while training
`--with_inverse_prompt` - Use Inverse Prompt while training
`--inverse_prompt_weight 0.1` - Weight of the inverse prompt for loss function
`--inverse_prompt_weight 0.1` - Weight of the inverse prompt for loss function
**Note:** The defaults in `train_prompting.sh` are the best performing values.
**Note:** The defaults in `train_prompting.sh` are the best performing values.
@ -197,6 +211,7 @@ Generate belief states by running the below script:
```shell
```shell
sh test_prompting.sh -m <tuned-prompt-model-path>
sh test_prompting.sh -m <tuned-prompt-model-path>
```
```
The argument `-m` takes the relative path of saved model from `SAVED_MODELS_PROMPT` env variable. It takes the following structure `-m <data-split-name>/<experiment-folder>/<epoch-folder>`
The argument `-m` takes the relative path of saved model from `SAVED_MODELS_PROMPT` env variable. It takes the following structure `-m <data-split-name>/<experiment-folder>/<epoch-folder>`