The Transformer architecture dominates NLP and vision. Its core operation — multi-head self-attention — involves $O(n^2 d)$ complexity for sequence length $n$ and dimension $d$. Quantum approaches aim to reduce this via: (a) quantum amplitude encoding of embeddings, (b) quantum inner product estimation for attention scores, and (c) quantum projection for key/query/value matrices.
Quixer (Khatri, Matos, Coopmans et al., 2024) implements a fully quantum self-attention mechanism using Linear Combination of Unitaries (LCU) and QSVT primitives to realise attention as a quantum operation. It is the first quantum transformer applied to a practical language modelling task (Penn Treebank).
Token embeddings $\mathbf{x}_1, \ldots, \mathbf{x}_n \in \mathbb{R}^d$ are loaded as quantum states:
The attention weight between tokens $i$ and $j$ requires computing $\mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d}$. Quixer uses the SWAP test to estimate this inner product:
More precisely, Quixer uses the Hadamard test with controlled-SWAP to extract signed inner products:
Input tokens: x₁, x₂, ..., xₙ ∈ ℝᵈ
│
▼ [QRAM loading — O(log d) depth]
|x₁⟩, |x₂⟩, ..., |xₙ⟩ (quantum states)
│
▼ [Quantum Q,K,V projection via PQC(θ)]
|q₁⟩, |k₁⟩, |v₁⟩, ... (projected states)
│
▼ [Quantum inner product estimation]
│ For all pairs (i,j):
│ aᵢⱼ = ⟨qᵢ|kⱼ⟩ via Hadamard test
│
▼ [Classical softmax — executed on classical CPU]
│ αᵢⱼ = softmax(aᵢⱼ / √d)ⱼ
│
▼ [Quantum linear combination of |vⱼ⟩ states]
│ |outᵢ⟩ = Σⱼ αᵢⱼ |vⱼ⟩ (quantum superposition)
│
▼ [Quantum measurement → classical output]
outᵢ ∈ ℝᵈ (attention-weighted embeddings)
│
▼ [Classical FFN + residual]
Next layer or output
vs. classical: $O(n^2 d)$. Advantage when $d \gg \text{poly}(\log d)$, i.e. for large embedding dimensions — contingent on efficient QRAM and fault-tolerant hardware.
HyQuT (Kong et al., Nov 2025) takes a hybrid approach: integrating VQCs into the Transformer at both 8M and 150M parameter scales. Just 10 qubits with 80 quantum gates replace ~10% of classical parameters in the 150M model while achieving comparable generation quality.
The quantum value projection uses an $\ell$-layer VQC with data re-uploading:
HyQuT Block:
Input X (N×d)
│
├──[Classical W_Q]──→ Q (N×d_k) ─┐
├──[Classical W_K]──→ K (N×d_k) ─┤─→ softmax(QKᵀ/√d_k) → α (N×N)
│ │ │
└──[Quantum VQC(θ)]──→ V (N×d_v) ←─┘ α · V → Output │
│ ↑ │
│ VQC internals: └────────────────┘
│ |0⟩──[Enc(x)]──[VQC layer 1]──...──[VQC layer ℓ]──[Meas]
│ Enc(x): angle encoding xᵢ → Ry(xᵢ) rotations
│ VQC layer: Ry(θ) + CNOT entangling
Classical FFN
│
▼
Layer Output
HyQuT uses data re-uploading to increase effective circuit depth without increasing qubit count:
This makes the VQC a universal function approximator on its input domain (Pérez-Salinas et al. 2020).
QTHA (Quantum Token-based Hybrid Attention) computes per-token quantum circuits to parameterise the compatibility function between queries and keys.
For token $i$, the quantum circuit $\mathcal{C}_i$ processes the query $\mathbf{q}_i$ and produces a quantum state encoding its "attention signature":
Attention weight between token $i$ (query) and $j$ (key):
For each query token i:
qᵢ ∈ ℝᵈ
│
▼ [Angle encoding]
|0⟩──[Ry(q_i1)]──[Ry(q_i2)]──...──[CNOT layers]──[Ry(θ)]──
│
┌──────────────┘
│ Quantum state |ψᵢ⟩
│
For each key token j: ▼
kⱼ ∈ ℝᵈ ──[encode into observable Mⱼ]── ⟨ψᵢ|Mⱼ|ψᵢ⟩ = aᵢⱼ
│
▼
aᵢⱼ for all j ──→ softmax ──→ αᵢⱼ
│
αᵢⱼ · Vⱼ ─────────────────────→ outputᵢ
| Model | Quantum Components | Q Complexity | NISQ Feasibility | Potential Advantage |
|---|---|---|---|---|
| Quixer | Full Q attention (QRAM + inner product) | $O(n^2 \log d)$ | Needs QRAM — not yet NISQ | Exponential in $d$ |
| HyQuT | Quantum V; Classical Q, K | $O(n \cdot \text{VQC cost})$ | NISQ compatible | Expressibility of V |
| QTHA | Per-token Q compatibility | $O(n \cdot \text{VQC cost})$ | NISQ compatible | Attention pattern richness |
| QViT | Quantum Q, K, V projections (orthogonal) | $O(n^2 \log d)$ circuit | NISQ compatible | $O(n)$ parameters |