ย
Blog Posts
Bash script to kill TPUv5e(TPUv5lite) consuming process
MLDL Framework
GCP
TPU
Mar 25, 2024
TPUv5 uses new device name, new scripts to kill the process :)
HF model โ OpenAI compatible API
NLP
MLDL Framework
Dev
Feb 26, 2024
transformers-openai-api๋ฅผ ํตํด ๋ช
๋ น์ด ํ์ค๋ก OpenAI Compatible API ๋ง๋ค๊ธฐ
gcloud ์ปค๋งจ๋๋ก VS Code SSH๋ก GCP VM ์ ์ํ๊ธฐ
GCP
Cloud
Ubuntu
Dev
Nov 30, 2023
๊ฐ๋จํ ProxyCommand๋ฅผ ํตํด์ ์ ์ํ ์ ์๋ ๋ฐฉ๋ฒ์ SSH config์ ์ค์ ํด์ฃผ์.
Huggingface Transformers Train with FSDP on PyTorch/XLA @ TPU
MLDL Framework
Dev
Cloud
TPU
Sep 9, 2023
Huggingface Transformers FSDP ์ฝ๋๋ก TPU์์ PyTorch/XLA๋ก ์ธ์ด๋ชจ๋ธ์ ํ์ตํด๋ณด์!
PyTorch/XLA SPMD @ TPU
MLDL Framework
Dev
GCP
Cloud
TPU
Sep 9, 2023
์๋ก์ด ๋ฐฉ์์ GSPMD๋ฅผ PyTorch/XLA(2.2)์์ ์จ๋ณด์. TPU๋ฅผ ์ง์ํ๋ค!
MLC LLM
NLP
MLDL Framework
Aug 22, 2023
์์ฒญ ๋น ๋ฅด๋ค๋ MLC LLM, ์๋นํด๋ณด์.
DeepSpeed Multinode
NLP
MLDL Framework
Dev
Aug 18, 2023
PySpark JSONL ๋ก๋ ๋๋ฆด๋ Schema ์ ๊ณต์ผ๋ก ์๋ ๋์ด๊ธฐ
MLDL Framework
Dev
Jul 14, 2023
StructType์ ํตํด์ Schema๋ฅผ ์ ๊ณตํด์ฃผ๋ฉด MetaData ๋ง๋ค๊ธฐ ์ํ ๋ก๋๋ฅผ ํด๊ฒฐํ ์ ์๋ค.
Numeric Values to Text, ์ซ์ ๋ฐ์ดํฐ๋ก ๋ ํ๋ฅผ ํ
์คํธ๋ก ์์ฑํ๊ธฐ
Preference Ranking Optimization for Human Alignment ๋
ผ๋ฌธ๋ฆฌ๋ทฐ
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
RLHF
Jul 3, 2023
RL ์์ด 1์์vs๋๋จธ์ง, 2์์vs๋๋จธ์ง, โฆ๋ก Human Preference ํ์ต์ํค๊ธฐ
Direct Preference Optimization ๋
ผ๋ฌธ๋ฆฌ๋ทฐ
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
RLHF
Jun 28, 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model ๋
ผ๋ฌธ๋ฆฌ๋ทฐ
ํน์ ํ Repo์ Branch๋ก pip install
Dev
Jun 26, 2023
pip install git+์ฃผ์ ์ Branch ์ง์ ํด ์ค์นํ๊ธฐ
RRHF ๋
ผ๋ฌธ & ์ฝ๋๋ฆฌ๋ทฐ: Rank Responses to Align Language Models with Human Feedback without tears
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
RLHF
Jun 23, 2023
์ฝ๋์ ํจ๊ปํ๋ RRHF ๋ฆฌ๋ทฐ
Cloudflare Tunnels ํ ํธ์คํธ์ ์ฌ๋ฌ๊ฐ ๋์ฐ๊ธฐ
Dev
Cloud
Jun 7, 2023
Docker๋ก Cloudflare Tunnel์ ๋์์ ํ ๊ธฐ๊ธฐ์ ์ฌ๋ฌ Cloudflare Tunnel์ ์ฐ๊ฒฐํ๊ธฐ
ElasticSearch Docker Single Node ์คํ ์คํจ ํด๊ฒฐ๋ฒ
Ubuntu
Dev
May 2, 2023
Elastic search error: "Native controller process has stopped - no new native processes can be started" ์๋ฌ ํด๊ฒฐ๋ฒ
PEFT๋ก LoRA Checkpoint ๋ก๋์ size mismatch ํด๊ฒฐ๋ฒ
MLDL Framework
NLP
PLM
Apr 3, 2023
base_model.model.gpt_neox.layers.0.attention.query_key_value.lora_A.weight: copying a param with shape torch.Size([16, 5120]) from checkpoint, the shape in current model is torch.Size([8, 5120]) ์ ๊ฐ์ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๊ธฐ
Synology NAS File Station์์ ๋ค์ด๋ก๋ ๋งํฌ๋ฅผ wget/curl๋ก ๋ฐ๋ ๋ฐฉ๋ฒ
NAS
Dev
Mar 25, 2023
Firefox Plugin โCLIGETโ์ผ๋ก ๋ค์ด๋ก๋ ๋งํฌ๋ฅผ ๋ฐ์ค์!
pix2pix-zero: Zero-shot Image-to-Image Translation ๋
ผ๋ฌธ๋ฆฌ๋ทฐ
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
CV
Feb 9, 2023
์ถ๊ฐ ํ์ต ์๋ ๊ณ ์ฑ๋ฅ์ Image To Image ๋ชจ๋ธ
datasets ๋ผ์ด๋ธ๋ฌ๋ฆฌ์ load_metric ์ฌ์ฉ์ Nonetype Error ๋ฐ์์ ํด๊ฒฐ๋ฒ
MLDL Framework
Dev
Ubuntu
Sep 21, 2022
TL;DR: ์บ์ ์ง์ฐ๊ณ scikit-learn์ ์ฌ์ค์นํ์
MaxMatch-Dropout/ Subword Regularization for WordPiece
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
COLING 2022
Sep 13, 2022
WordPiece์ Subword Dropout์ ์ ์ฉํ์!
K-MHaS: A Multi-label Hate Speech Detection Dataset in Korean Online News Comment
NLP
COLING 2022
TLDR๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Sep 8, 2022
์๋ก์ด ํ๊ตญ์ด HateSpeech Dataset!
Ordinal Log Loss: A simple log-based loss function for ordinal text classification
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
COLING 2022
Sep 8, 2022
Ordinal classification๋ฅผ ์ํ ๊ฐ๋จํ๊ณ ์ฑ๋ฅ ์ข์ Loss function
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
NLP
CV
PLM
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Sep 5, 2022
๋ช์ฅ(3~5๊ฐ)์ ์ด๋ฏธ์ง ๋ง์ผ๋ก Diffusion ๋ชจ๋ธ์ ์๋ก์ด ์บ๋ฆญํฐ๋ฅผ ๋ฑ์ฅ์ํค์!
nvidia-smi๊ฐ ๋๋ฌด ๋ง์ cpu usage ๋ณด์ผ๋
Ubuntu
Dev
Aug 8, 2022
nvidia-smi๊ฐ ๋๋ฌด ๋ง์ cpu usage ๋ณด์ผ๋, nvidia-smi daemon์ ์ค์ ํ์.
Word-Level Fine-Grained Story Visualization
NLP
CV
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Aug 5, 2022
Prompt Tuning for Generative Multimodal Pretrained Models
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
NLP
TLDR๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Aug 5, 2022
FP16์ผ๋ก Transformers Pipeline์ ๋ชจ๋ธ ๋ก๋ํ๊ธฐ
MLDL Framework
NLP
Aug 4, 2022
Python Time-Ordered UUID
Dev
Aug 4, 2022
Caddy๋ก Reverse Proxy HTTPS ์๋นํ๊ธฐ (feat. Letโs encrypt)
Dev
Ubuntu
Feb 28, 2022
Huggingface Transformers Pipeline
NLP
MLDL Framework
Feb 10, 2022
Huggingface Transformers ๋ผ์ด๋ธ๋ฌ๋ฆฌ์ pipeline์ ์ฌ์ฉํ๋ ์ต์
๋ค
Ubuntu 21.04์ Mecab-ko ์ค์น ์ค apt ๊ด๋ จ ์ค๋ฅ ๋ฐ์์
NLP
Ubuntu
Feb 8, 2022
TL;DR: automake๋ฅผ ์๋์ผ๋ก ์ค์นํด์ฃผ๋ฉด ๋๋ค.
gsutil ๋ค์ด๋ก๋ ์๋ฃ ์๋ ๊ฒฝ์ฐ
GCP
Cloud
Feb 7, 2022
GCP TPU VM์์ gsutil์ ํตํ ๋ค์ด๋ก๋๊ฐ 99%์์ ์คํจํ ๊ฒฝ์ฐ ํด๊ฒฐ๋ฒ
FUDGE: Controlled Text Generation With Future Discriminators
NLP
TLDR๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Jul 22, 2021
KcBERT-v2022, KcELECTRA-v2022
NLP
์ฌ์ด๋ํ๋ก์ ํธ
PLM
Feb 7, 2022
2022๋
๋ฒ์ ์ KcBERT์ KcELECTRA
KcT5 Pretraining on TPU (feat. Flax)
NLP
์ฌ์ด๋ํ๋ก์ ํธ
PLM
Feb 8, 2022
ํ๊ตญ์ด ๋๊ธ๋ก TPUv3-8์์ T5 ์ฌ์ ํ์ตํ๊ธฐ with Flax, Jax
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Jun 25, 2021
BERT์์ Word Emb, Pos(Relative) Emb๋ฅผ ์ชผ๊ฐ ๋ ๋ฒกํฐ๋ก ๊ฐ๊ฐ ๊ณ์ฐํ์!
ZeRO-Infinity
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
MLDL Framework
May 30, 2021
DeepSpeed ZeRO-Infinity
FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
NLP
TLDR๋
ผ๋ฌธ๋ฆฌ๋ทฐ
May 24, 2021
ICLR2021, PLM(BERT)์ ์ถ๊ฐ ๋ชจ๋ ๋ถ์ด๊ณ , Contrastive learning + Regualizer๋ก Debiased๋ output ์ถ์ถํ๋ ๋ฐฉ๋ฒ๋ก .
Transformers Trainer ๋ฏ์ด๋ณด๊ธฐ
NLP
MLDL Framework
May 22, 2021
Huggingface Transformers ํ์ต Wrapper, Trainer๊ฐ ์ด๋ป๊ฒ ๋์ํ๋์ง ์์๋ณด์!
Docker + DeepSpeed + MultiGPU ์ฌ์ฉ ์ค NCCL posix_fallocate failed: No space left on device ์๋ฌ ๋์ํ๊ธฐ
NLP
MLDL Framework
May 20, 2021
๋์ปค ์ปจํ
์ด๋ ์์์ DeepSpeed + MultiGPU ์ฌ์ฉ์, NCCL No Space left on device ์๋ฌ๊ฐ ๋ฐ์ํ๋ ๊ฒฝ์ฐ์ ํด๊ฒฐ์ฑ
Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
TLDR๋
ผ๋ฌธ๋ฆฌ๋ทฐ
CV
May 24, 2021
ViT์์ Transformer Attention์ ๋จ์ํ FF Layer๋ก ๋ฐ๊ฟจ๋๋ฐ ์ฑ๋ฅ์ด ๋น์ท. 79.9(ViT) vs 77.9(FF Layer only)
Transformers์ DeepSpeed๋ก ์ BERT๋ชจ๋ธ ๊ตฝ๊ธฐ
NLP
MLDL Framework
May 17, 2021
Transformers
run_mlm.py
์ DeepSpeed, ZeRO-2/ZeRO-3์ผ๋ก ์ BERT ๊ตฝ๊ธฐHuggingface + DeepSpeed + FairScale
NLP
MLDL Framework
May 16, 2021
Huggingface๋ก 'ํฐ' ๋ชจ๋ธ ํ์ตํ๊ธฐ
DExperts: On-the-Fly Controlled Text Generation with Experts and Anti-Experts
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
May 14, 2021
Language Model Finetune ํตํด Detoxify & Sentiment Controlled Generation ํ๊ธฐ
Transformers ์ ๋ชจ๋ธ ๋ง๋ค๊ธฐ
NLP
MLDL Framework
May 14, 2021
๐คHuggingface Transformers์ ์๋ก์ด ๋ชจ๋ธ ๊ตฌ์กฐ๋ฅผ ๋ง๋ค์ด๋ณด์!
exBERT: Extending Pre-trained Models with Domain-speci๏ฌc Vocabulary Under Constrained Training Resources
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Mar 19, 2021
๊ธฐ์กด BERT์ ์๋ก์ด Vocab & (์๋์ ์ผ๋ก)์์, ๋ณ๋ ฌ BERT๋ชจ๋ธ์ ๋ถ์ฌ์ ํ์ต์, Domain Adaptation(DAPT)๊ฐ ์์ฃผ ์ ๋๋ค! (์ฝ 5-6%p์ ๊ท ์ผํ ์ฑ๋ฅ ํฅ์์ ๋ณด์)
Longformer
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
Mar 27, 2021
BERT max len 512๋ฅผ ๋์ด 4096๊น์ง, Sequence length์ O(n)์ธ Attention Transformer
Cite.GG
์ฌ์ด๋ํ๋ก์ ํธ
May 12, 2021
๋ณด๋ค ์ฌ์ด <์ฝ์ ๋
ผ๋ฌธ๊ฑฐ๋ฆฌ ์ฐพ๊ธฐ>๋ฅผ ์ํด, Cite.GG
GeDi: Generative Discriminator Guided Sequence Generation
NLP
๋
ผ๋ฌธ๋ฆฌ๋ทฐ
May 1, 2021
GPT 110M์ผ๋ก GPT-2(XL, 1.2B), GPT-3(175B) Generation Guideํ๊ธฐ
Train Language Model on TPU
NLP
May 11, 2021
TPU๋ก Language Model ํ์ตํด ๋ณด์! ๐ฅ
ย
ย
about
ย