GELU vs ReLU: The Smooth Gate Inside GPT
ReLU cuts every negative activation to zero. GELU computes a smooth tanh-shaped gate; I unpack the formula one real number at a time and place it inside GPT's feed-forward network.
Field Notes
A working notebook.
Always in progress.
Notes on event-driven Java, Spring Boot, Kafka, and secure systems for regulated banking.
Latest
LayerNorm from Four Numbers: Mean, Variance and a Token Row
I refused to let LayerNorm remain a formula. Starting with four integers, I work through mean, variance, standard deviation, normalization and the exact PyTorch operation used for each GPT token.
A Gentle Introduction to PyTorch: Tensors Before Attention
Before I could make sense of attention code, I needed PyTorch's tiny tensor vocabulary to stop feeling mysterious. This first session covers shapes, axes, batching, and the important difference between stack, cat, multiplication, and dot products.
The hoodie stays: turning my mascot into a reusable Codex skill
A generated character only becomes an identity when it survives the next prompt. I turned one navy-hoodie mascot, two successful references and a --manga switch into a small reusable Codex skill.
After Daybreak: enabling Codex Security on code I own
I applied for verified Daybreak access, completed the identity check, and mapped the next step: connecting my own GitHub repositories to Codex Security without giving up human review or scope control.
LLMs, the whole thing · Part 9
Multi-head attention, one shape at a time
Part 9 of LLMs, the whole thing: two heads, one token, and the difference between computing attention and organizing its tensors. A concrete PyTorch walkthrough of view, transpose, concatenation and the output projection.
Systems in Motion: a visual identity of my own
A new identity for my portfolio, writing, and independent products: graphite, electric blue, and a moving Z inspired by event-driven systems. The design choices, reusable article templates, and engineering constraints behind it.
Running the box: Docker, Linux, the JVM
Cleaning up your systemd service
Retiring a service is more than killing the process. Here's the four-beat teardown I ran to remove a honeypot from my box — and why 'disable' isn't 'stop'.
LLMs, the whole thing · Part 8
Causal attention and the two masks: one hides the future, one cuts links on purpose
Section 3.5 of Raschka's book on seven real GPT-2 tokens: why masking after softmax breaks the row sums, why -inf on the scores fixes it in one pass, and how dropout is a second, random mask that doubles its survivors.
I audited my prompts for a model that reads literally
Running Anthropic's prompt-audit over 2,700 lines of Claude Code steering before moving to Fable 5.1. The grep for pressure language hit once. The cruft was drift and changelog phrasing, and seven of the eight recommended prompt snippets were already in the harness.
Start here
The multi-part series and running themes, rather than one long pile.
-
Series
LLMs, the whole thing
Working through large language models from the ground up — turning text into tokens, embeddings and the geometry of meaning, the transformer architecture, attention, how these models are trained, and what separates one generation from the next.
-
Series
Building agents on the JVM with Embabel
Learning Embabel by building with it, in order — planning, durable sessions, testing without a model, tools, human-in-the-loop, tracing, exposing it over MCP.
-
Series
Shipping an AI feature into my own product
Build it, attack it, keep extending it — the Ask Zakaria concierge from the first prompt through prompt-injection red-teaming to reading a job description and taking questions by voice.
-
Topic
Traces, Tempo and the LGTM stack
Instrument it, learn to query it, then point it at something that thinks.
-
Topic
Working with Claude Code
An agent as a collaborator rather than an autocomplete — design systems, durable knowledge, automation ladders, subagent fleets on my own source, and now in CI.
-
Topic
Working with Codex / ChatGPT
Using Codex and ChatGPT as engineering collaborators — repository workflows, security reviews, automation, and the practical boundaries that keep the work controlled.
-
Topic
Security and privacy, in practice
Attacking my own features and reading the protocols underneath them — secrets delivery, response headers, a published address for bad news, a cookie banner that sets no cookies, and what an auth flow is actually protecting you from.
-
Topic
Building this site in public
Shipping and operating my own product in public — one jar serving two hosts, making it findable, dropping experiments into it, shipping a free CV builder off the side.
Browse by tag
- #agents
- #aggregation
- #ai
- #ai-agents
- #ai-tooling
- #anthropic-skills
- #appsec
- #architecture
- #artifacts
- #attention
- #automation
- #banking
- #batch
- #bean-validation
- #bert
- #binding
- #blackboard
- #branching
- #branding
- #broadcasting
- #causal-attention
- #cgroups
- #chatgpt
- #ci-cd
- #classpath
- #claude
- #claude-code
- #cli
- #codex
- #configuration
- #cosine-similarity
- #csp
- #data-modeling
- #dataviz
- #daybreak
- #ddd
- #deep-learning
- #deepseek
- #design-system
- #developer-experience
- #developer-tooling
- #developer-tools
- #devex
- #devops
- #dns
- #docker
- #dropout
- #embabel
- #embeddings
- #events
- #fable
- #fail2ban
- #fairness
- #fastapi
- #fastmail
- #fine-tuning
- #flyway
- #from-java-to-python
- #frontend
- #fundamentals
- #gdpr
- #gelu
- #git
- #github-actions
- #goap
- #gpt
- #grafana
- #guardrails
- #hibernate-validator
- #honeypot
- #hooks
- #http
- #human-in-the-loop
- #ideas
- #image-generation
- #java
- #java-to-python
- #jinja2
- #junit
- #jvm
- #kafka
- #keycloak
- #knowledge-base
- #layer-normalization
- #learning
- #linux
- #llm
- #llms
- #llms-txt
- #loki
- #loops
- #lora
- #machine-learning
- #maths-for-llms
- #mcp
- #mockito
- #mongodb
- #multi-agent
- #multi-head-attention
- #networking
- #nlp
- #oauth
- #oauth2
- #observability
- #okf
- #ollama
- #openkb
- #opentelemetry
- #ops
- #otel
- #outbox
- #parking
- #patterns
- #pgvector
- #positional-encoding
- #postgres
- #print-css
- #privacy
- #prompt-audit
- #prompt-injection
- #pydantic
- #python
- #pytorch
- #rag
- #react
- #refactoring
- #relu
- #rocm
- #search-console
- #secrets
- #security
- #self-attention
- #seo
- #shipping
- #skills
- #spring
- #spring-ai
- #spring-batch
- #spring-boot
- #spring-data
- #spring-data-jdbc
- #sse
- #ssh
- #state
- #systemd
- #templating
- #tempo
- #tensors
- #testing
- #tokenization
- #tools
- #traceql
- #tracing
- #traefik
- #training-data
- #transactions
- #transformers
- #tts
- #typescript
- #unsloth
- #uv
- #vectors
- #vite
- #waitfor
- #web-speech-api
- #workflow
- #worktree