Added Transformer document

README.md

@@ -1,7 +1,7 @@
-## Update (3/26/2025) 
+## Update (3/28/2025) 
 
-LSTM & GRU support has been added.
-Check out new example workflows at `examples/workflows/lstm_classification.json` & `examples/workflows/gru_classification.json`. Both of them achieve approximately **87% validation accuracy** for IMDB sentiment classification (which is in the expected range for LSTM & GRU).
+Transformer support has been added.
+Check out the new example workflow at `examples/workflows/embedding_transformer_classification.json`. This workflow achieves approximately 83% validation accuracy on the IMDB sentiment classification task using the Hugging Face version of the dataset, which is within the expected range for training a Transformer from scratch with limited data.
 
 ---
 # ComfyUI-Pt-Wrapper  

docs/model_training.md

@@ -101,4 +101,7 @@ If you want to specify a loss function for training instead of using the default
 Refer to `examples/workflows/train_model_with_custom_loss_example.json` for an example. (This workflow is intended to demonstrate node connections and is not optimized for high CIFAR10 accuracy. It runs for only 10 epochs to allow a quick check and reaches 88.9% accuracy.) You should adjust the hyperparameters according to your specific goals.
 
 # Training RNN
-Refer to [Training RNN for text classification](training_rnn_for_classification.md)
+Refer to [Training RNN for text classification](training_rnn_for_classification.md)
+
+# Training Transformer
+Refer to [Training Transformer for text classification](training_transformer_for_classification.md)

docs/training_rnn_for_classification.md

@@ -18,7 +18,7 @@ The flow consists of the following groups of nodes:
 
 ## Loading Data
 
-Specify a Hugging Face text dataset using **Ptf Hf Glove Dataset** to load the text data, and encode each word using GloVe embeddings.  
+Specify a Hugging Face text dataset using **Ptv Hf Glove Dataset** to load the text data, and encode each word using GloVe embeddings.  
 Then use **Pt Data Loader** to batch the data and feed it into the training node.
 
 ## Setting Up Optimizer and Scheduler

docs/training_transformer_for_classification.md

@@ -0,0 +1,45 @@
+# Training Transformer for Text Classification
+
+You can use **ComfyUI-Pt-Wrapper** to train a Transformer for text classification, such as sentiment analysis on a movie review dataset.
+
+You can find the workflow at `examples/workflows/embedding_transformer_classification.json`.
+
+The flow consists of the following groups of nodes:
+* Loading data
+* Setting up the model, loss, optimizer, and scheduler
+* Training
+
+## Loading Data
+
+This phase consists of two steps:
+* Specify the SentencePiece encoder for converting text to token IDs
+* Specify a Hugging Face text dataset to load
+
+## Specify SentencePiece encoder for converting text to token IDs
+Use Sp Load Model and Sp Encode to load the SentencePiece model and specify the encode function for text to token ID convertion.
+
+## Specify a Hugging Face text dataset to load
+Specify **Ptv Hf Dataset With Token Encode** to load the text data, and encode each word into token IDs using the Sp Encode function above.
+
+Then use **Pt Data Loader** to batch the data and feed it into the training node.
+
+## Setting Up Optimizer and Scheduler
+
+This workflow uses **Pto AdamW** as the optimizer and **Pto Lr Scheduler Cosine Annealing** as the scheduler. You can also experiment with other optimizers and schedulers included in this extension.
+
+## Loss
+
+Use **Ptn BCE with Logits Loss**, which includes a built-in sigmoid activation, allowing you to pass raw logits directly from the model.
+
+## Model
+
+Use an Transformer model with a linear head (**Ptn Embedding Transformer Linear**). This model contains the embedding layer at the bottom.  Note that this embedding layer is not pretrained, so we are training this layer from scratch in this workflow.
+
+## Training
+
+Specify **Pt Train Transformer Classification Model** and configure training parameters.  
+Set the vocabulary size to **32000** to match the SentencePiece tokenization model.
+
+## Expected Accuracy
+
+Using this workflow, you should achieve around 83% validation accuracy on the Hugging Face version of the IMDB dataset. While this may seem low to some readers, it's important to note that the model is trained entirely from scratch, with no pretrained layers (aside from tokenization to IDs). Given that, this accuracy is within the expected range.