Russian language processing via Spark(NLP) 🔥

Micro project on big data technologies via spark

Content:

1. Colab-Spark setup

2. Data loading

3. EDA & Preprocessing

4. Pipelines & Experiments

5. Text preprocessing

6. Text classification

   • BoW models + LogReg
   • Transfer Learning (at least an attempt 😀)

7. Entity Recgnition & Entity Linking

Tech stack:

• pySpark & pySpark.ml
• sparkNLP
• pymorphy2
• SpaCy
• HuggingFace
• Textblob

...and much more 🤘