GELU vs ReLU: The Smooth Gate Inside GPT
ReLU cuts every negative activation to zero. GELU computes a smooth tanh-shaped gate; I unpack the formula one real number at a time and place it inside GPT's feed-forward network.
Tag
#relu
- #agents
- #aggregation
- #ai
- #ai-agents
- #ai-tooling
- #anthropic-skills
- #appsec
- #architecture
- #artifacts
- #attention
- #automation
- #banking
- #batch
- #bean-validation
- #bert
- #binding
- #blackboard
- #branching
- #branding
- #broadcasting
- #causal-attention
- #cgroups
- #chatgpt
- #ci-cd
- #classpath
- #claude
- #claude-code
- #cli
- #codex
- #configuration
- #cosine-similarity
- #csp
- #data-modeling
- #dataviz
- #daybreak
- #ddd
- #deep-learning
- #deepseek
- #design-system
- #developer-experience
- #developer-tooling
- #developer-tools
- #devex
- #devops
- #dns
- #docker
- #dropout
- #embabel
- #embeddings
- #events
- #fable
- #fail2ban
- #fairness
- #fastapi
- #fastmail
- #fine-tuning
- #flyway
- #from-java-to-python
- #frontend
- #fundamentals
- #gdpr
- #gelu
- #git
- #github-actions
- #goap
- #gpt
- #grafana
- #guardrails
- #hibernate-validator
- #honeypot
- #hooks
- #http
- #human-in-the-loop
- #ideas
- #image-generation
- #java
- #java-to-python
- #jinja2
- #junit
- #jvm
- #kafka
- #keycloak
- #knowledge-base
- #layer-normalization
- #learning
- #linux
- #llm
- #llms
- #llms-txt
- #loki
- #loops
- #lora
- #machine-learning
- #maths-for-llms
- #mcp
- #mockito
- #mongodb
- #multi-agent
- #multi-head-attention
- #networking
- #nlp
- #oauth
- #oauth2
- #observability
- #okf
- #ollama
- #openkb
- #opentelemetry
- #ops
- #otel
- #outbox
- #parking
- #patterns
- #pgvector
- #positional-encoding
- #postgres
- #print-css
- #privacy
- #prompt-audit
- #prompt-injection
- #pydantic
- #python
- #pytorch
- #rag
- #react
- #refactoring
- #relu
- #rocm
- #search-console
- #secrets
- #security
- #self-attention
- #seo
- #shipping
- #skills
- #spring
- #spring-ai
- #spring-batch
- #spring-boot
- #spring-data
- #spring-data-jdbc
- #sse
- #ssh
- #state
- #systemd
- #templating
- #tempo
- #tensors
- #testing
- #tokenization
- #tools
- #traceql
- #tracing
- #traefik
- #training-data
- #transactions
- #transformers
- #tts
- #typescript
- #unsloth
- #uv
- #vectors
- #vite
- #waitfor
- #web-speech-api
- #workflow
- #worktree