Testing a Claude Code plugin against no plugin at all
I built a small commit-message plugin and ran claude plugin eval on it. The no-plugin baseline showed what the plugin actually adds, and the one red case turned out to be the judge's mistake.
Tag
#evals
- #accessibility
- #adt
- #agents
- #aggregation
- #ai
- #ai-agents
- #ai-architecture
- #ai-tooling
- #anorm
- #anthropic
- #anthropic-skills
- #applicative
- #appsec
- #architecture
- #artifacts
- #attention
- #automation
- #banking
- #batch
- #bean-validation
- #bert
- #binding
- #blackboard
- #blame
- #branching
- #branding
- #broadcasting
- #buildcv
- #building-in-public
- #cake-pattern
- #calibration
- #case-classes
- #cats-core
- #causal-attention
- #cgroups
- #chatgpt
- #chatgpt-sites
- #ci
- #ci-cd
- #cka
- #classpath
- #claude
- #claude-api
- #claude-code
- #claude-sdk-scala
- #cli
- #code-review
- #codex
- #collections
- #concurrency
- #configuration
- #cosine-similarity
- #cross-entropy
- #csp
- #data-modeling
- #dataviz
- #daybreak
- #ddd
- #deep-learning
- #deepseek
- #dependency-injection
- #design-system
- #design-systems
- #developer-experience
- #developer-tooling
- #developer-tools
- #devex
- #devops
- #dns
- #docker
- #domain-modeling
- #dropout
- #either
- #embabel
- #embeddings
- #enterprise-integration-patterns
- #enumeratum
- #eq
- #error-handling
- #evals
- #events
- #fable
- #fail2ban
- #fairness
- #fastapi
- #fastmail
- #fine-tuning
- #flyway
- #focus
- #foldmap
- #for-comprehensions
- #from-java-to-python
- #frontend
- #full-text-search
- #functional-programming
- #functor
- #fundamentals
- #gdpr
- #gelu
- #git
- #github-actions
- #goap
- #gpt
- #gradient-descent
- #gradients
- #grafana
- #gsap
- #guardrails
- #guice
- #haskell
- #hibernate-validator
- #honeypot
- #hooks
- #http
- #human-in-the-loop
- #hyperframes
- #ideas
- #identity-brokering
- #image-generation
- #implicit-classes
- #implicits
- #intellij
- #java
- #java-interop
- #java-to-python
- #jev
- #jinja2
- #junit
- #jvm
- #kafka
- #keycloak
- #kind
- #knowledge-base
- #kubernetes
- #layer-normalization
- #learning
- #linux
- #llm
- #llm-agents
- #llm-costs
- #llms
- #llms-from-scratch
- #llms-txt
- #loki
- #loops
- #lora
- #machine-learning
- #maths-for-llms
- #maven
- #mcp
- #migration
- #mockito
- #modernization
- #mods
- #mongodb
- #monoid
- #multi-agent
- #multi-head-attention
- #networking
- #nlp
- #oauth
- #oauth2
- #observability
- #oidc
- #okf
- #ollama
- #openkb
- #opentelemetry
- #ops
- #option
- #opus
- #opusplan
- #otel
- #outbox
- #parking
- #pattern-matching
- #patterns
- #patterns-for-engineers
- #pentesting
- #pgvector
- #play
- #play-framework
- #playframework
- #plugins
- #positional-encoding
- #postgres
- #postgresql
- #pretraining
- #print-css
- #privacy
- #probability
- #prompt-audit
- #prompt-caching
- #prompting
- #prompt-injection
- #pydantic
- #python
- #pytorch
- #rag
- #react
- #refactoring
- #relu
- #resilience
- #rocm
- #sbt
- #scala
- #scalafmt
- #scala-play
- #scalatest
- #sealed-trait
- #search-console
- #secrets
- #security
- #self-attention
- #semigroup
- #seo
- #shipping
- #skills
- #softmax
- #sonnet
- #specs
- #spring
- #spring-ai
- #spring-batch
- #spring-boot
- #spring-data
- #spring-data-jdbc
- #sse
- #ssh
- #state
- #ste100
- #subagents
- #svg
- #systemd
- #technical-writing
- #templating
- #tempo
- #tensors
- #testing
- #tokenization
- #tooling
- #tools
- #traceql
- #tracing
- #traefik
- #training-data
- #traits
- #transactions
- #transformers
- #tts
- #type-classes
- #typescript
- #unsloth
- #uv
- #validation
- #vectors
- #video
- #visualizations
- #vite
- #voice-cloning
- #waitfor
- #web-design
- #webgpu
- #web-speech-api
- #workbench
- #workflow
- #workflows
- #worktree
- #wsl
- #xss