Copy of exBERT - Code ver.

exBERT at Github

cgmhaicenter/exBERT

The details of the model is in paper. In command line: python Pretraining.py -e 1 -b 256 -sp path_to_storage -dv 0 1 2 3 -lr 1e-04 -str exBERT -config path_to_config_file_of_the_OFF_THE_SHELF_MODEL ./config_and_vocab/exBERT/bert_config_ex_s3.json -vocab ./config_and_vocab/exBERT/exBERT_vocab.txt -pm_p path_to_state_dict_of_the_OFF_THE_SHELF_MODEL -dp path_to_your_training_data -ls 128 -p 1 You can replace the path_to_config_file_of_the_OFF_THE_SHELF_MODEL and path_to_state_dict_of_the_OFF_THE_SHELF_MODEL to any weel pre-trained model in BERT archietecture../config_and_vocab/exBERT/bert_config_ex_s3.json defines the size of extension module.

https://github.com/cgmhaicenter/exBERT

• 스타는 겨우 9개..

• issue 올라온거 처리 안해줌. (1달 전에 올라온건데.)

실험로그

다운로드 & 커맨드 실행

git clone https://github.com/cgmhaicenter/exBERT

이슈1. 패키지 설치 안되어있음

pip install torch transformers tqdm numpy

이슈2. Github의 커맨드는 엔터 \ 가 없음..

python Pretraining.py -e 1 
  -b 256 
  -sp path_to_storage
  -dv 0 1 2 3 -lr 1e-04 
  -str exBERT    
  -config path_to_config_file_of_the_OFF_THE_SHELF_MODEL ./config_and_vocab/exBERT/bert_config_ex_s3.json  
  -vocab ./config_and_vocab/exBERT/exBERT_vocab.txt 
  -pm_p path_to_state_dict_of_the_OFF_THE_SHELF_MODEL
  -dp path_to_your_training_data
  -ls 128 
  -p 1

이슈3. path_to_config_file_of_the_OFF_THE_SHELF_MODEL 없음

python Pretraining.py -e 1 \
  -b 256 \
  -sp path_to_storage \
  -dv 0 1 2 3 -lr 1e-04  \
  -str exBERT     \
  -config beomi/kcbert-base ./config_and_vocab/exBERT/bert_config_ex_s3.json   \
  -vocab ./config_and_vocab/exBERT/exBERT_vocab.txt \
  -pm_p beomi/kcbert-base \
  -dp path_to_your_training_data \
  -ls 128  \
  -p 1