This course will teach you the theory behind large language models (LLMs) and give you the tools to implement LLMs in your applications.

Contact

• Dr. Bruno Yun

  • Email: [email protected]
  • Website: https://bruno-yun.notion.site/
  • Office: Nautibus, 12.038 (second floor).
  • Working hours: 9:00 am to 5:00 pm

  Send your queries by email. For a face-to-face meeting, contact me by email to arrange one.

Structure of the course

This part of the course will have 18 hours of lectures, 12 hours of practicals.

The teaching timetable is split into three sessions as follows.

• Part 1: Zoom on the transformer architecture (3 hours, 12th of November 2024)

  The RNN Architectures

  The Transformer Model for NLG

  Continuous assessment 1 (written test, 30%) → Deadline 01/12/2024 23:59

  Student project submissions

• Part 2: A short history of large language models (3 hours, 18th of November 2024)

  A quick survey of existing LLMs

  Running LLMs on Consumer-Grade Computers

• Part 3: Pre-training and Post-training (3 hours, 25th of November 2024)

  Pre-training

  Post-training/Fine-tuning

• Part 4: Practical techniques and tools for large language models (3 hours, 2nd of December 2024)

  Prompt engineering

  Retrieval Augmented Generation

  Useful tools for large language models

  Group for CC2 and CC3

  Continuous assessment 2 (oral presentation, 30%) → Deadline 10th of January 2025

  Practical 0: Background on PyTorch (if needed)

  Practical 1: Tokenizers, embeddings, and multi-head attention from scratch (3 hours, 2nd of December 2024)

• Part 5: Evaluation and ethical questions of large language models (3 hours, 9th of December 2024)

  Evaluating LLMs

  Security, privacy, and ethical questions of LLM

• Part 6: Exciting research of large language models (3 hours, 16th of December 2024)

  Pros and Cons of selective state space models (SSMs)

  Test-time training (TTT) models

  Mixture-of-experts

  Large Language Models in 1.58 Bits?

• Part 7-9: Practicals

  Practical 2: Implementing GPT2 and pre-training (3 hours, 10th of January 2025)

  Practical 3: Fine-tuning GPT2 and simple LLM-based evaluation (3 hours, 17th of January 2025)

  Practical 4 Building applications with LLMs (3 hours, 24th of January 2025)

  Continuous assessment 3 (code and report, 40%) → Deadline 26th of January 2025

Recommended reading (among others)

Reading list