Create an environment for prompt-based methods (Python 3.10 is **required**)
```shell
@ -48,6 +49,7 @@ To deactivate the conda environment, run: (Only after running all the experiment
```shell
conda deactivate
```
#### Download and extract SOLOIST pre-trained model
Download and unzip the pretrained model, this is used for fine-tuning the baseline and prompt-based methods. For more details about the pre-trained SOLOIST model, refer to the GitHub [repo](https://github.com/pengbaolin/soloist).
@ -66,10 +68,12 @@ Clone the repository for source code
The generated predictions are saved under `OUTPUTS_DIR_BASELINE` folder. Some of the generated belief state predictions are uploaded to this repository and can be found under [outputs](outputs) folder.
### Baseline Evaluation
@ -148,6 +155,7 @@ Edit the [evaluate.py](baseline/evaluate.py) to set the predictions output file
```shell
python evaluate.py
```
### Results from baseline experiments
|data-split| JGA |
|--|:--:|
@ -172,20 +180,26 @@ Change directory to `prompt-learning` and install the requirements. Make sure th
cd prompt-learning
pip install requirements.txt
```
### Train the prompt model
Train a separate model for each data split. Edit the [train_prompting.sh](prompt-learning/train_prompting.sh) file to modify the default hyperparameters for training (learning rate, epochs).
```shell
sh train_prompting.sh -d <data-split-name>
```
Pass the data split name to `-d` flag.
Possible values are: `5-dpd`, `10-dpd`, `50-dpd`, `100-dpd`, `125-dpd`, `250-dpd`
Example training command: `sh train_baseline.sh -d 50-dpd`
**Some `train_prompting.sh` flags**:
`--num_epochs 10` - Number of epochs
`--learning_rate 5e-5` - Initial learning rate for Optimizer
`--with_inverse_prompt` - Use Inverse Prompt while training
`--inverse_prompt_weight 0.1` - Weight of the inverse prompt for loss function
**Note:** The defaults in `train_prompting.sh` are the best performing values.
@ -197,6 +211,7 @@ Generate belief states by running the below script:
```shell
sh test_prompting.sh -m <tuned-prompt-model-path>
```
The argument `-m` takes the relative path of saved model from `SAVED_MODELS_PROMPT` env variable. It takes the following structure `-m <data-split-name>/<experiment-folder>/<epoch-folder>`