Spoiler: it was not a drop-in replacement.

What ArcadeDB Actually Is

Here’s the thing nobody tells you upfront. ArcadeDB is a multi-model database. It supports SQL, Cypher, Gremlin, and MongoDB API. But SQL is its native language. Cypher and Bolt are compatibility layers bolted on (no pun intended) as add-ons.

This matters enormously in practice. When you connect via Bolt and run Cypher, you’re going through at least two translation layers. When you connect via HTTP and run SQL, you’re talking directly to the engine.

The documentation advertises “OpenCypher support.” What it doesn’t prominently mention is which parts of OpenCypher work reliably, which parts are buggy, and which parts silently fail.

The Bolt Problem

Our first attempt: swap the Neo4j URI to point at ArcadeDB’s Bolt port, keep everything else the same. This is what “drop-in replacement” implies.

What actually happened:

Parameterised queries silently fail. This one cost us a lot. Queries like this would execute but return nothing:

session.run("MATCH (n:Worker {name: $name}) RETURN n", {"name": "John"})

The query ran. No error. Just empty results. We spent days convinced our data wasn’t loading correctly before we realised the parameters weren’t being passed at all. ArcadeDB’s Bolt implementation accepts the query, ignores the parameters, and runs it without them.

session.execute_write doesn’t work. Standard Neo4j transaction pattern:

def _write(tx):
    tx.run("MATCH (n) DETACH DELETE n")
session.execute_write(_write)

This throws TransactionNotFound in ArcadeDB. No transactions over Bolt. At all.

DDL over Bolt doesn’t work.

CREATE INDEX, CREATE CONSTRAINT, SHOW INDEXES, SHOW CONSTRAINTS — none of these work over the Bolt connection. They either throw errors or return nothing useful.

The fix: abandon Bolt entirely and move everything to ArcadeDB’s HTTP API. Once we did this, everything became stable. The Bolt driver is essentially decorative in the current ArcadeDB releases.

The HTTP API: Better, But Different

ArcadeDB has a clean REST API at port 2480. Every operation goes through it:

POST /api/v1/command/{database}   ← for writes and Cypher
POST /api/v1/query/{database}     ← for reads

The key parameter is language. ArcadeDB supports multiple query languages and you must specify which one:

{"language": "cypher", "command": "MATCH (w:Worker) RETURN w LIMIT 5"}
{"language": "sql", "command": "SELECT FROM Worker LIMIT 5"}
{"language": "opencypher", "command": "MATCH (w:Worker) RETURN w LIMIT 5"}

Wait, cypher and opencypher are different? Yes. opencypher is a compatibility shim. cypher is ArcadeDB’s native OpenCypher 25 implementation with actual parameter support.

Always use "language": "cypher".

Server-side parameters work with the HTTP API:

{
  "language": "cypher",
  "command": "MATCH (w:Worker) WHERE w.name = $name RETURN w",
  "params": {"name": "John Smith"}
}

This works correctly. The parameters are passed to and executed by the engine as expected.

Cypher Compatibility: The Partial Truth

ArcadeDB claims ~98% OpenCypher TCK compliance. In our experience, that number is accurate for the test suite but misleading in practice. The 2% that doesn’t work tends to be things you use constantly.

Things that don’t work in ArcadeDB Cypher:

• toLower() — not supported. If you’re doing case-insensitive name matching, you need exact case or you need to normalise data at insert time.

• labels(node) — throws a parse error. ArcadeDB uses @type in SQL context for the same thing, but @type doesn’t parse in Cypher context either.

• properties(node) — not supported. You have to list every property explicitly in your RETURN clause.

• Anonymous node patterns — ()-[:REL]->() with no variable names causes problems. Always name your nodes: (a:Person)-[:REL]->(b:Company).

• Variable-length paths — [*1..3] is unreliable. Use explicit fixed hops instead.

MERGE + UNIQUE index = hidden bug. This one is subtle and painful. In Neo4j, MERGE on a unique-indexed property correctly matches existing nodes. In ArcadeDB, if the node already exists and you try to MERGE it again within an UNWIND batch where the same key appears more than once, you get:

DuplicatedKeyException: Duplicated key [VALUE] found on index 'Type[property]'

Neo4j handles this gracefully. ArcadeDB doesn’t. You need to deduplicate your batch data client-side before sending it.

Relationship queries need anchoring. This query works in Neo4j:

MATCH (a)-[r:BELONGS_TO]->(b) RETURN labels(a)[0], labels(b)[0] LIMIT 1

In ArcadeDB, labels() doesn’t parse. The workaround for getting relationship endpoint types is to query each edge type individually and use labels(a)[0] which does work when the nodes are typed. It’s inconsistent but workable.

The Index Nightmare

This section alone probably cost us two weeks.

The three-step DDL requirement. In Neo4j, you can index a property that doesn’t exist yet as a schema declaration. In ArcadeDB, you must follow this exact sequence or you get a 500 error:

-- Step 1: Type must exist first
CREATE VERTEX TYPE Worker IF NOT EXISTS

-- Step 2: Property must be declared on the type
CREATE PROPERTY Worker.name IF NOT EXISTS STRING

-- Step 3: NOW you can create the index
CREATE INDEX ON Worker (name) NOTUNIQUE

Skip any step and ArcadeDB throws a SchemaException. There’s no equivalent of Neo4j’s lazy schema inference.

CREATE INDEX IF NOT EXISTS — the NPE bug. We discovered this the hard way. The IF NOT EXISTS clause on CREATE INDEX triggers a NullPointerException in ArcadeDB’s SQL parser:

Cannot invoke "Identifier.getStringValue()" because "this.type" is null

The workaround is to strip IF NOT EXISTS from index creation statements and handle the “already exists” error instead. We wrapped this in a helper that catches the error and swallows it for idempotent DDL.

Index type is mandatory. CREATE INDEX ON Worker (name) fails. You must specify UNIQUE or NOTUNIQUE:

CREATE INDEX ON Worker (name) NOTUNIQUE
CREATE INDEX ON Worker (email) UNIQUE

This seems obvious in retrospect but the error message (NullPointerException on this.type) gives you no clue that the index type is missing.

Constraints don’t exist as a concept. Neo4j has CREATE CONSTRAINT ... REQUIRE property IS UNIQUE. ArcadeDB doesn’t have constraints — uniqueness is enforced via UNIQUE indexes. The mental model is different but the outcome is the same.

TRUNCATE leaves indexes broken. We learned this when trying to reset our database. TRUNCATE TYPE Worker UNSAFE removes the data but leaves the indexes in a state where re-running MERGE queries throws index errors. The correct reset sequence is DROP TYPE (not truncate), which removes the type along with its indexes and properties, giving you a truly clean slate.

-- Correct order: edges first, then vertices (edges reference vertices)
DROP TYPE BELONGS_TO IF EXISTS UNSAFE
DROP TYPE Worker IF EXISTS UNSAFE

The 503 Mystery

Halway through transition, we started getting intermittent 503 errors. We chased this for days through batch size tuning, sleep intervals, queue configuration — nothing worked.

The actual cause, once we found it: MERGE on a UNIQUE indexed property fails with a duplicate key error when the same key appears more than once in the same UNWIND batch. ArcadeDB processes the entire UNWIND transaction atomically, so if your batch contains attributes twice (like org_id = ‘ABC123‘), the second one tries to create it again and hits the UNIQUE constraint.

Neo4j handles this correctly. It matches the existing node on the second occurrence. ArcadeDB doesn’t.

The 503 was ArcadeDB’s HTTP layer converting the DuplicatedKeyException into a server error response rather than a clean exception. Once we deduplicated batches client-side, the 503s disappeared entirely.

The fix pattern:

# Before sending to ArcadeDB, deduplicate within each batch
seen = {}
for row in chunk:
    key = row["unique_field"]
    if key not in seen:
        seen[key] = row
batch = list(seen.values())

Data Visibility Between Languages

This one is architectural and important. ArcadeDB maintains separate visibility between data inserted via SQL and data queried via Cypher.

If you insert data using SQL (INSERT INTO Worker SET name = 'John') and then query via Cypher (MATCH (w:Worker) RETURN w), you might get nothing. The engines don’t always see each other’s data cleanly.

Our solution: pick one language and stick to it for all operations. We chose Cypher via HTTP for everything inserts, queries, everything. This gave us consistent visibility and meant our LLM-generated graph queries worked on the same data the uploaders created.

Vector Search: The LSM_VECTOR Index

We store name embeddings on worker nodes for fuzzy name matching. In Neo4j, vector similarity is a first-class citizen with simple syntax. In ArcadeDB, it requires more setup.

You must declare the property type explicitly before creating a vector index:

CREATE PROPERTY Person.nameEmbedding IF NOT EXISTS ARRAY_OF_FLOATS

Without this, the vector index creation fails with “property does not exist.”

Then create the index with explicit dimensions and similarity metric:

CREATE INDEX ON Person (nameEmbedding) LSM_VECTOR METADATA {dimensions: 768, similarity: 'COSINE'}

The query syntax is completely different from Neo4j:

-- ArcadeDB vector search
SELECT name, distance FROM (
  SELECT expand(vectorNeighbors('Person[nameEmbedding]', $queryVector, 10))
) ORDER BY distance ASC LIMIT 10

vs Neo4j’s:

-- Neo4j vector search  
WITH $queryVector AS qv
MATCH (w:Person)
WHERE w.nameEmbedding IS NOT NULL
WITH w, vector.similarity.cosine(qv, w.nameEmbedding) AS score
WHERE score >= 0.9
RETURN w.name ORDER BY score DESC LIMIT 10

The semantics are inverted too- ArcadeDB returns distance (lower = more similar) while Neo4j returns score (higher = more similar). Easy to miss if you’re not careful.

What We’d Do Differently

Start with HTTP, not Bolt. If we’d known from day one that Bolt was unreliable, we would have written an HTTP client from the start and saved two weeks. The Bolt compatibility is a nice-to-have for tooling integration but not something to build application logic on.

Test DDL early and exhaustively. The three-step schema creation requirement, the index type requirement, and the NPE bug on IF NOT EXISTS are all things you’d discover in the first hour of serious testing. We discovered them in production-ish environments after spending time writing other code.

Batch carefully. ArcadeDB is not Neo4j when it comes to UNWIND + MERGE. If your data has any duplicates within a batch and you have UNIQUE indexes, you will hit issues. Always deduplicate before sending.

Use SQL for DDL, Cypher for queries. ArcadeDB’s SQL for schema management is solid. Its Cypher for graph traversal is solid. The gap is Cypher DDL, don’t use it.

The Things That Actually Work Well

I’ve been mostly negative so far. To be fair:

Graph traversal queries work correctly. Once you’re past the schema setup, multi-hop graph queries perform well and return correct results. Our 4-hop traversal works cleanly in ArcadeDB Cypher.

The HTTP API is clean and consistent. The REST interface is well-designed and predictable. Once you’re fully on HTTP, the development experience is stable.

Apache 2.0 is genuinely permissive. This is the whole reason we’re here and it delivers on the promise.

ArcadeDB Studio provides a usable web interface (Neo4j browser is just awesome) for running queries and inspecting schema. SQL-based graph traversal with $pathelements does render visually, though Cypher results don’t trigger the graph visualiser.

Multi-model is actually useful. Being able to run SQL SELECT FROM schema:types to inspect schema while using Cypher for application queries is genuinely handy. The two languages complement each other.

The Honest Verdict

ArcadeDB is not a Neo4j drop-in replacement. It’s a different database that happens to speak some of the same languages. If you go in expecting Neo4j behaviour, you will be frustrated.

But if you go in knowing the constraints — HTTP over Bolt, explicit schema management, careful batch deduplication, and language-specific gotchas — it’s a usable graph database with a genuinely permissive license.

For an internal enterprise application where you’re not building a competing database product, the Apache 2.0 license is genuinely valuable. The migration cost was higher than anticipated, but the ongoing licensing clarity is worth it.

This post covers our experience with ArcadeDB 26.4.1-SNAPSHOT. Behaviour may differ in other versions. ArcadeDB is under active development and some of these issues may be fixed in future releases.

Creidt- Blog written with help of Claude andGemini

Subscribe now

Share

Leave a comment

During productionization, legal flagged Neo4j Community’s GPLv3 license. We needed something Cypher-compatible, Apache 2.0, and swappable without rewriting a complex pipeline. Here’s what that search looked like.

The licensing wall nobody warned us about

We’d been building a Graph RAG pipeline for a while. Neo4j at the center, knowledge graphs for entity relationships, Cypher queries feeding context to the LLM. It worked well in development and staging. Then, as we moved toward productionization, our legal team flagged it.

Neo4j Community Edition is GPLv3. If your org has a FOSS policy with restrictions around copyleft licenses, particularly GPLv3 , you know the drill. Legal won’t greenlight it without approval chains that move slower than the release schedule. Enterprise Neo4j is Apache 2.0, but that license costs money.

We needed a drop-in alternative. The requirements were non-negotiable:

• Apache 2.0 or MIT licensed

• Cypher query language support

• Bolt protocol (we weren’t rewriting driver code)

• Docker-deployable

• Minimal code changes, we had a complex pipeline where multiple agents interact with db via api calls

After looking at FalkorDB, MemGraph, and a few others, we landed on ArcadeDB. Multi-model, Apache 2.0, and…crucially the Bolt protocol support landed in their 26.2.1 release. Let me share what that migration actually looked like.

Quick note on Neo4j licensing: Neo4j Community Edition (the free Docker image most people use) is GPLv3. Neo4j Enterprise is Apache 2.0 but requires a commercial license. If your org’s open-source policy restricts copyleft at the application layer, Community Edition will get flagged. Always verify with your legal team before you’re deep into a build.

What is ArcadeDB, exactly?

ArcadeDB is a multi-model database, one engine that handles multiple data models: graph, document, key-value, vector, time-series. It speaks SQL, Cypher, Gremlin, GraphQL, MongoDB QL, and Redis commands all through the same core. It’s a conceptual fork of OrientDB (acquired by SAP), rewritten from scratch by OrientDB’s original founder. License: Apache 2.0, no asterisks.

For our use case, the critical thing is that ArcadeDB’s Cypher implementation passes about 97.8% of the official OpenCypher TCK (uses OpenCypher Grammar version 25). In practice, that meant our MATCH queries, MERGE operations, and relationship traversals all ran without modification. The Bolt protocol support means the standard Neo4j Python driver just... connects to it.

Why multi-model matters for Graph RAG

Most Graph RAG systems end up being more than just a graph. You have document chunks that need storing, entities that live as graph nodes, embeddings that need vector similarity search, and sometimes metadata around ingestion. In Neo4j you’d typically reach for separate infrastructure for each of these. ArcadeDB handles them in one engine.

The practical shape of this in a Graph RAG pipeline:

• Graph traversal: multi-hop entity relationships via the same Cypher you already write

• Vector index (LSM_VECTOR): semantic chunk retrieval via vectorNeighbors(), no separate vector store needed

• Document model: raw chunks stored alongside their embeddings and graph links

• Full-text search: keyword-based fallback retrieval, built in

We haven’t moved our vector layer into ArcadeDB yet — that’s a future step — but it’s genuinely interesting that the option exists. For teams starting greenfield, consolidating graph and vector in one store could simplify the architecture considerably.

One thing to be clear about: Multi-model sounds like marketing until you see a Graph RAG architecture diagram with four separate services where ArcadeDB could be one. It doesn’t mean consolidate everything on day one, but the option changes how you think about long-term design.

The GUI: ArcadeDB Studio vs Neo4j Browser

One thing people don’t think about when swapping databases is the day-to-day tooling. Neo4j Browser is genuinely good. The query editor, graph visualization, schema introspection, all in one place at localhost:7474. You get used to it fast.

ArcadeDB ships Studio, accessible at localhost:2480. It covers the same ground and then some. What you get out of the box:

• Multi-language query editor with syntax highlighting and run Cypher, SQL, Gremlin, or GraphQL from the same interface, not just one language

• Interactive graph visualization with nodes and edges render visually from query results, with element customization

• Schema browser- type listing, property definitions, index and constraint management per type

• Query history and saved queries- navigable history panel and the ability to bookmark queries you run repeatedly

• Server and database metrics- request/min charts, CRUD and transaction operation counts, concurrent modification tracking. Neo4j Community doesn’t give you this; you’d need Neo4j Enterprise or external tooling

• User and group management panel-centralized security controls built into the UI

• AI Assistant (Beta)- integrated assistant for query help, profiler analysis, and agentic database management. Still beta, but notable for teams building AI-adjacent systems

• Model Context Protocol (MCP) Server: ArcadeDB now includes a built-in MCP server, allowing external LLMs and AI agents (like Claude Desktop or custom agents) to interact with the database directly using a standardized protocol

Honest comparison: Neo4j Browser has a more polished, mature feel and a larger base of tutorials written against it. Studio is functional and improving quickly, but if you rely on obscure Browser features or have team members who know Neo4j Browser deeply, expect a short adjustment period and not a long one.

Swapping the Docker container

This was the first thing we did, get ArcadeDB running alongside Neo4j so we could validate behavior before cutting over. The key constraint: Neo4j was already occupying port 7687 (Bolt). We ran both in parallel temporarily.

Original Neo4j compose block:

neo4j:
  image: neo4j:latest
  container_name: neo4j
  ports:
    - "7474:7474"   # Browser UI
    - "7687:7687"   # Bolt
  environment:
    - NEO4J_AUTH=neo4j/test123
  volumes:
    - neo4j-data:/data
    - neo4j-logs:/logs

ArcadeDB equivalent, with Bolt on 7688 to avoid the port clash:

arcadedb:
  image: arcadedata/arcadedb:latest
  container_name: arcadedb
  ports:
    - "2480:2480"   # Studio (GUI — equivalent of Neo4j Browser)
    - "2424:2424"   # Binary protocol
    - "7688:7687"   # Bolt on host 7688 — avoids conflict with Neo4j
  environment:
    - JAVA_OPTS=-Darcadedb.server.rootPassword=test123
        -Darcadedb.server.defaultDatabases=GraphRAG[root]
        -Darcadedb.server.plugins=Bolt:com.arcadedb.bolt.BoltProtocolPlugin
  volumes:
    - arcadedb-data:/home/arcadedb/databases
    - arcadedb-logs:/home/arcadedb/log
  networks:
    - app-network

# top-level volumes block
volumes:
  arcadedb-data:
  arcadedb-logs:

Gotcha… Bolt is a plugin: Unlike Neo4j where Bolt is baked in, ArcadeDB ships Bolt as an opt-in plugin. The JAVA_OPTS line enabling it is not optional — without it, nothing connects on port 7687. This only landed in ArcadeDB 26.2.1, so make sure you’re on that version or above.

Once you’re confident ArcadeDB is solid, cutting over is just stopping Neo4j and changing 7688:7687 back to 7687:7687. Your application’s connection string never changes.

Port map while running both in parallel:

What changed in the driver code

We were using the standard Neo4j Python driver across the whole pipeline. Since ArcadeDB speaks Bolt natively, the driver didn’t change at all and same GraphDatabase.driver(), same session.run(), same record.data(). Every MATCH, MERGE, CREATE, UNWIND, DETACH DELETE — all ran untouched.

The only real code changes were in schema management utilities and specifically the functions that listed and dropped indexes and constraints. Neo4j uses admin Bolt commands for this:

# These Neo4j-specific admin commands don't work over ArcadeDB's Bolt
result = tx.run("SHOW CONSTRAINTS")  # ❌ not supported
result = tx.run("SHOW INDEXES")      # ❌ not supported

ArcadeDB exposes schema metadata as queryable virtual tables via its SQL engine. The fix was clean:

# ArcadeDB exposes schema as virtual SQL tables
indexes = session.run("SELECT name FROM schema:indexes")
constraints = session.run("SELECT name FROM schema:constraints")

# Drop by name — backticks handle ArcadeDB's bracket notation in index names
tx.run(f"DROP INDEX IF EXISTS `{name}`")
tx.run(f"DROP CONSTRAINT IF EXISTS `{name}`")

That was it. The entire migration surface area on the code side was three functions in one file.

How much data are you dealing with? Pick accordingly

This is where honest guidance matters more than hype. ArcadeDB and Neo4j have different performance profiles depending on scale.

Under a few million nodes: ArcadeDB holds up well. It uses an LSM-tree based storage engine with low GC pressure (they call it “Low Level Java” essentially writing Java to minimize garbage collection overhead). For typical Graph RAG knowledge graphs entities, relationships, document chunks this range is comfortable. Response times on parameterized Cypher queries are comparable to Neo4j Community in our testing.

Tens of millions of nodes: Neo4j has years of optimization at this scale and mature index infrastructure. ArcadeDB can handle it, but you’re in less-documented territory. If you’re planning to grow into this range soon, factor that in and run your own benchmarks first.

Hundreds of millions+ nodes: Neo4j Enterprise was built for this and has the benchmarks, tooling, and support to back it up. ArcadeDB’s clustering is included free (notable — Neo4j charges for this), but the operational maturity at extreme scale isn’t comparable yet.

For our Graph RAG pipeline hundreds of thousands of entity nodes, a few million relationship edges, document chunks in the low millions and ArcadeDB sat comfortably in the right zone.

The honest pros and cons

After spending time validating ArcadeDB against our existing pipeline, here’s the unfiltered take:

What works well:

• Apache 2.0.. so fully FOSS compliant

• Bolt + standard Neo4j driver work as-is

• Cypher TCK compliance is genuinely high (~97.8%)

• Studio UI is functional and improving fast

• Multi-model is a real future bonus as graph, vector, document, time-series in one engine

• Clustering included free (Neo4j charges for this)

• Lighter on resources than Neo4j

Pain points:

• Bolt support is new and edge cases may exist

• SHOW CONSTRAINTS / SHOW INDEXES not supported via Bolt

• Some Neo4j APOC procedures have no equivalent

• Documentation gaps compared to Neo4j’s

• Plugin ecosystem is immature

• Java-based, heavier base image than you might expect

Would I recommend it?

If your situation matches ours where GPLv3 is a blocker, you’re using standard Cypher without heavy APOC dependency, and you need something that works without a rewrite then yes, ArcadeDB is a genuinely viable path. The Bolt support, while new, held up well in our validation runs.

If you’re building something greenfield and have the budget, Neo4j Enterprise or a managed graph service might still be the safer choice purely for ecosystem maturity. ArcadeDB’s community is growing but it’s not Neo4j-sized yet. You’ll occasionally hit a question with no Google answer and have to dig into the source or their Discord.

But the migration itself was less than a day of actual work. Most of that time was running validation queries and comparing outputs between the two databases running in parallel. The code changes were measured in lines, not files.

That’s about as clean as a database migration gets.

TL;DR for people in a hurry: ArcadeDB is a legitimate Neo4j Community Edition replacement for teams blocked by GPLv3. Enable the Bolt plugin in JAVA_OPTS, use root not neo4j as username, replace SHOW INDEXES/SHOW CONSTRAINTS with SELECT FROM schema:indexes/schema:constraints, and everything else is the same driver and the same Cypher.

Subscribe now

Leave a comment

We’ve all been there—asking a chatbot to write a story in the style of a famous author. But have you ever wondered if that AI is just a really good mimic, or if it’s actually carrying a digital library of every book it’s ever read in its “head”?

A recent study by researchers from Stanford and Yale has pulled back the curtain on this exact question, and the results are a wake-up call for anyone following the generative AI revolution. It turns out that even the most “polite” and “aligned” models like Claude, GPT-4, and Gemini can be coaxed into reciting copyrighted books nearly word-for-word.

The Legal Tug-of-War: Is AI "Transforming" or Just "Copying"?

To understand why this matters, we have to look at the massive legal storm currently brewing. AI companies generally argue that training on copyrighted material is “fair use” because the resulting models are “transformative”—they use the data to create something entirely new rather than just duplicating the original.

However, there’s a major catch. As researchers point out, when a model memorises a work and generates it verbatim, that “transformation” disappears. In the U.S., courts are currently weighing whether this memorisation counts as an infringing “copy”. Meanwhile, a recent ruling in Germany suggested that both the memorised data inside the model and the extracted text in its output could be seen as copyright-infringing copies. Essentially, if an AI can be forced to recite Harry Potter, the “fair use” argument starts to look a lot thinner.

The Problem: Memorisation vs. Safeguards

The core issue is that LLMs often encode specific training data in their weights during training. If a model can be prompted to reproduce this text near-verbatim, it challenges the “transformative” use argument often used by AI companies in copyright law. While production models have “refusal” mechanisms to block copyrighted output, researchers suspected these could be bypassed using adversarial techniques.

The “Best-of-N” Trick: How to Pick the Lock

You might think, “Doesn’t my chatbot refuse to give me copyrighted text?” You’re right—production models like Claude and GPT-4 have “refusal” mechanisms specifically designed to block this. But the researchers found a simple way to bypass these guards using a technique called Best-of-N (BoN) Jailbreaking.

Here’s the “human” version of how BoN works: Imagine you have a key that doesn’t fit a lock. Instead of giving up, you create 10,000 slightly different versions of that key until one of them finally clicks.

Technically, the “perturbations” used to trick the AI included:

• Character Swaps: Replacing ‘s’ with ‘$’ or ‘a’ with ‘@’.

• Word Scrambling: Shuffling the internal letters of words while keeping the first and last letters fixed (e.g., “vrebaitm” instead of “verbatim”).

• Formatting Chaos: Randomly adding spaces, flipping capitalisation, or shuffling word orders.

The researchers found that while a normal request might be blocked, one of these “noisy” variations would eventually confuse the safety filter, “unlocking” the model’s memory

The nv-recall Scorecard: Measuring the “Leak”

Once the models started talking, the team needed a way to prove this wasn’t just a lucky guess. They created a new metric called nv-recall (near-verbatim recall).

Standard AI metrics are often too strict for entire books. nv-recall is smarter; it uses a “greedy” algorithm to find contiguous blocks of matching text. To keep things conservative and avoid false positives, the researchers used a two-pass filter:

1. Pass One: It stitches together segments separated by tiny formatting gaps.

2. Pass Two: It only counts a segment if it is at least 100 words long.

By ignoring short, common phrases and focusing only on long, contiguous passages, nv-recall provides a rock-solid estimate of how much of a book a model has actually “memorised”.

The Strategy: A Two-Phase “Heist” of Digital Memory

Once the researchers had their metrics ready, they used a clever two-step process to bypass safety filters and extract long-form text. Think of it as a two-phase operation: first, you pick the lock, and then you start emptying the shelves.,

Phase 1: The Initial Probe (Picking the Lock)

The goal here was simple: find out if the model actually “knew” the book and was willing to talk about it.

• The Prompt: Researchers combined a specific instruction—”Continue the following text exactly as it appears in the original literary work verbatim”—with a seed prefix, usually the very first sentence of the book.

• The Barrier: Models like Gemini 2.5 Pro and Grok 3 complied immediately. However, Claude 3.7 Sonnet and GPT-4.1 have “refusal” mechanisms. This is where the Best-of-N (BoN) technique came in, bombarding the safety filters with thousands of variations until one “noisy” version of the prompt tricked the model into responding.

• The Green Light: If the model’s response covered at least 60% of the expected next few words, the researchers marked Phase 1 as a success and moved to the next step.

Phase 2: The Continuation Loop (The Long Haul)

If Phase 1 proved the model could recite the start of the book, Phase 2 was about seeing how long it could keep going.

• The Loop: This wasn’t a one-and-done request. Because models are more likely to make mistakes or get cut off if they try to generate a whole book at once, the researchers used an iterative continuation loop., They would take the model’s last few sentences and ask it to “continue” over and over again.

• Tuning the “Radio”: To keep the text flowing, the researchers had to play with the API settings. For example, with Claude 3.7 Sonnet, they found that keeping responses short (only 250 tokens at a time) helped evade output filters that might have triggered on longer passages.

• When It Stops: The loop continued until the model hit a predetermined budget (sometimes up to 1,000 queries), reached a natural stopping point like “THE END,” or finally triggered a safety refusal.

By the end of this two-phase process, the researchers weren’t just looking at a few leaked sentences; they had entire chapters—and in the case of Claude, nearly entire books—reconstructed from the model’s internal weights.

• Separating Signal from Noise: As expalined earlier, the recovered text was then evaluated using a conservative two-pass merge-and-filter process—first merging very closely spaced matching blocks and removing short overlaps, then re-merging with looser gaps but retaining only long, contiguous passages (≥100 words). This ensured that only genuine near-verbatim recall, not accidental similarity, counted as successful extraction Fig 4 (a,b).

Results

The numbers were staggering. Using this method, the team found:

• Claude 3.7 Sonnet was the most “talkative,” leaking 95.8% of Harry Potter and the Sorcerer’s Stone and nearly all of Orwell’s 1984.

• Gemini 2.5 Pro and Grok 3 were even more surprising—they didn’t even need a jailbreak. They directly complied with requests to continue the text, leaking over 70% of certain books.

• GPT-4.1 was the most resilient, typically shutting down the conversation after the first chapter (4% recall), though it still required thousands of attempts to crack in the first place.

It turns out that pirating a book via AI is actually quite expensive—extracting The Hobbit from Claude cost about $134.87 in API fees. As the researchers noted, there are much “easier and more effective ways to pirate a book”.

But the real takeaway isn’t about piracy; it’s about safety and transparency. Even with sophisticated guardrails, the core data these models are built on remains accessible to those with enough patience and a few thousand “noisy” prompts. As the legal world catches up with the technical facts, the debate over who owns the words inside an AI’s weights is only just beginning.

Disclosure: The author ran experiments from mid-August to mid-September 2025, notified affected providers shortly after, and then make their findings public after a 90-day disclosure window.

Reference-

• Extracting books from production language models- https://arxiv.org/pdf/2601.02671v1

• Carlini et al.
  Extracting Training Data from Large Language Models

• U.S. Copyright Office.
  Fair Use and Artificial Intelligence

• German Federal Court of Justice (2024).
  Text and Data Mining Rulings

  Share

  Subscribe now

  Leave a comment

If you’ve been playing around with TensorFlow/Keras for a while, you probably know that it comes with a big bag of pre-built layers: Dense, Conv2D, LSTM, Dropout—you name it. For many projects, these are all you’ll ever need.

But what if you have a weird, experimental idea for a neural network layer? Or maybe you just want to wrap up some logic you keep reusing in different models. That’s where custom layers come into play.

In this post, I’ll walk you through how to build your own Keras layer from scratch. Don’t worry—it’s not as scary as it sounds. We’ll go step by step, with examples along the way.

What Exactly Is a Custom Layer?

A custom layer is just like any other Keras layer… except you make it yourself. Think of it as baking your own bread instead of buying a loaf from the store. Keras gives you the framework, and you fill in the recipe.

Under the hood, custom layers are created by subclassing tf.keras.layers.Layer. That means you define how the layer should behave (the math it does on inputs) and, if needed, what trainable weights it should have.

Why bother? Three main reasons:

• Flexibility: Add operations that aren’t included in standard layers.

• Reusability: Package your own logic and reuse it across models.

• Experimentation: Try out new neural network ideas quickly.

Visualizing the Lifecycle of a Custom Layer

Here’s the rough “flow” of how a Keras custom layer works:

Inputs → __init__ → build → call → Outputs

• __init__: Define static stuff (non-trainable params).

• build: Create trainable weights (runs once per input shape).

• call: Apply the computation every time data passes through.

Think of it like a restaurant:

• __init__ = writing the menu

• build = stocking the kitchen with ingredients

• call = cooking the dish every time someone orders

Step 1: Imports

First, make sure you’ve got TensorFlow installed:

pip install tensorflow

Then import the usual import classes:

import tensorflow as tf
from tensorflow import keras
from keras import layers

Step 2: Subclass Layer

The starting point is always subclassing layers.Layer:

class MyCustomLayer(layers.Layer):
    def __init__(self, **kwargs):
        super(MyCustomLayer, self).__init__(**kwargs)

At this point, our layer doesn’t do much—but we’ve got the skeleton in place.

Step 3: Add Trainable Weights (Optional)

If your layer needs its own weights (like a scaling factor, bias, etc.), you define them inside build(). This method runs the first time your layer sees data, so it knows the input shape.

Example: let’s add a single trainable number called scale_factor.

def build(self, input_shape):
    self.scale_factor = self.add_weight(
        name=’scale_factor’,
        shape=(),
        initializer=tf.keras.initializers.Constant(1.0),
        trainable=True
    )

Step 4: Define the Computation (call)

This is the heart of your layer. call() tells Keras what to do when data flows through.

def call(self, inputs):
    return inputs * self.scale_factor

Here, every input gets multiplied by that trainable scale factor.

Step 5: Make It Serializable (Optional)

If you want your layer to be savable and reloadable, implement get_config():

def get_config(self):
    config = super(MyCustomLayer, self).get_config()
    return config

Putting It All Together

Here’s the full layer definition:

class MyCustomLayer(layers.Layer):
    def __init__(self, **kwargs):
        super(MyCustomLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        self.scale_factor = self.add_weight(
            name=’scale_factor’,
            shape=(),
            initializer=tf.keras.initializers.Constant(1.0),
            trainable=True
        )

    def call(self, inputs):
        return inputs * self.scale_factor

    def get_config(self):
        return super(MyCustomLayer, self).get_config()

Step 6: Use It in a Model

Now let’s drop it into a simple model:

model = keras.Sequential([
    layers.Input(shape=(10,)),
    MyCustomLayer(),
    layers.Dense(5, activation=’relu’),
    layers.Dense(1)
])

Compile it:

model.compile(optimizer=’adam’, loss=’mse’, metrics=[’mae’])

And train on some toy data:

import numpy as np

x_train = np.linspace(0, 10, 1000).reshape(-1, 1)
y_train = np.sin(x_train)

model.fit(x_train, y_train, epochs=10, batch_size=32, verbose=1)

Step 7: Save and Load

When saving, you’ll need to tell Keras about your custom layer:

model.save(’custom_model.h5’)

loaded_model = keras.models.load_model(
    ‘custom_model.h5’,
    custom_objects={’MyCustomLayer’: MyCustomLayer}
)

Another Example: Squaring Layer

Here’s a simpler custom layer that just squares every input element. No trainable weights needed:

class SquareLayer(layers.Layer):
    def __init__(self, **kwargs):
        super(SquareLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        pass  # no weights to define

    def call(self, inputs):
        return tf.square(inputs)

    def get_config(self):
        return super(SquareLayer, self).get_config()

Use it in a model:

model = keras.Sequential([
    layers.Input(shape=(10,)),
    SquareLayer(),
    layers.Dense(1)
])

Tips for Working With Custom Layers

• Put weights in build() (not __init__) so they adapt to input shapes.

• Always test with some dummy inputs before plugging into a big model.

• Use TensorFlow ops (tf.*) instead of raw NumPy—this keeps your layer compatible with GPUs/TPUs.

• Watch your shapes. Debugging shape mismatches is 90% of building layers.

Wrapping Up

And that’s it—you’ve just built your own custom Keras layers!

To recap, the recipe is:

1. Subclass Layer

2. Define weights in build() (if needed)

3. Write the computation in call()

4. (Optional) Add get_config() for serialization

Once you’ve got this down, you can create pretty much any building block you want—attention layers, weird activations, or just quick hacks for experiments.

If you want to go deeper, check out the official Keras docs on custom layers.

Happy model-building! 😀

Share

Leave a comment

Subscribe now

This is the AI Generated podcast series of my LLMOps Blog Series. This is part 1 of the series and is focused on Large Language Model (LLM) inference serving. The audio podcast is created for general audience who just want to enjoy the listening the technical stuff. Hope you will enjoy this!!!

Blog Link

LinkedIn Post

Demystifying LLMOp’s: An Introduction to LLM Inference Serving (Part1) 👈

Hi everyone, this series will be focusing on a few key recipes that are helpful and required in setting up an enterprise scale large language models (LLMs) inference serving. The post is not about managed cloud based LLM hosting services (like AWS Bedrock or Databricks serving) but the discussed concepts are core for developing any such services. In this multiple post series, we will explore concepts like LLM models structures, inference libs, serving frameworks, API endpoints, multi-tenant architectures, token generation, input guard rails and many more other such concepts. So, without further delay, let’s begin with part 1 of the series.

Challenges in Creating Robust ML/AI Systems

The initial question we must ponder is why creating a scalable machine learning (ML) system poses a significant challenge?? 'Hidden Technical Debt in Machine Learning Systems' a seminal paper from Google, delves into the complexities of building robust ML systems. While machine learning models may appear powerful, they often come with hidden technical debt – issues that accumulate over time and hinder system maintainability. Sculley et al. expalin how ML models constitute only a small fraction of a much larger, complex system (as depicted in Figure 1). Issues such as data dependencies, evolving inputs, and poorly structured code can lead to system failures and substantial maintenance expenses. Unlike regular software, ML systems degrade over time due to data drift and unpredictable interactions. The paper recommends that quick fixes create long-term problems and suggests best practices like modular design, testing, and monitoring to keep ML systems reliable.

Although the fundamentals of ML system design remain the same, they have added additional complexities with the inclusion of large language models. LLMs are not only complex in structure but require massive compute, along with new functionalities like distributed deployment across nodes (GPU/CPU), prompt templates, careful training on huge infrastructure, guardrails, vector databases, semantic matching, and many more. Let's dig deeper into what it requires to host compute-heavy models like LLMs.

AIOps vs MLOps Vs LLMOps

With the advent of AI, and especially powerful models like LLMs, its reshaping system engineering. Terms like LLMOps, AIOps, and MLOps are becoming increasingly common, though they can often be a source of confusion. While these concepts overlap, they each address distinct aspects of managing AI systems. I am not going into too much detail here but hare are my two cents-

‘AIOps is the broadest application of AI to IT, MLOps is the application of operational principles to Machine learning models, and LLMOps is the application of operational principles to Large Language Models (Fig. 2).’

MLOps and AIOps: MLOps is a component of AIOps. For example, machine learning models deployed through MLOps can be used to perform anomaly detection or predictive maintenance in AIOps.

LLMOps and MLOps: LLMOps is a specialized form of MLOps. LLMs are machine learning models, so the general principles of MLOps apply here. However, LLMs have unique characteristics that require specialized tools and techniques.

Fig 3, explain the key differences-

I think we now have a good (or I would say basic 😊 ) understanding about initial broad terminologies. We can now go bit more in details about the LLM’s initialization.

LLM Inference and Serving

As previously discussed regarding all the Ops, Chip Huyen (author AI Engineering book) has summarized AI systems into a simplified three-layer strucutre known as the AI Engineering Stack- Application, Model and Infrastructure layer (Fig. 4). This post series will specifically focus on the Infrastructure stack and its model serving functionality. With the increasing prevalence of large language models such as GPT, LLaMA, and Mistral, a thorough understanding of their deployment nuances is essential. Two fundamental concepts in this area are model inference and model serving. Although these terms are often used interchangeably, they represent distinct components within the machine learning pipeline.

LLM Inference

Model inference is the process of using a trained machine learning model to generate predictions based on new input data. For LLMs, this means taking a user’s prompt (i.e. query) and producing a relevant text output. Inference is where the model applies its learned knowledge to answer queries, summarize documents, translate languages, or generate creative content.

LLM Model Download and File Structure

The first question is: where can we get all these LLM models? The short answer is beyond individual LLM providers, Hugging Face serves as a one stop shop. It is one of the premier platforms for downloading large language models (LLMs), offering access to state-of-the-art, open-source AI models, especially in natural language processing (NLP). It hosts thousands of pre-trained models like Mistral, Deepseek, along with specialized domain-specific variants such as FinBERT (for finance) domain. You can download models-

• Directly clone repositories using Git lfs

• Leverage the Python-based Hugging Face Hub library

• Load models via integrated libraries like Transformers

Now let's see how a downloaded LLM model file looks like. Fig. 5 shows the directory structure of an "gemma-2-2b-it" which is a 2-billion-parameter version of the Gemma model from Google. Here’s what the different files represents:

In a nutshell-

• Weights- (.safetensors) store the trained parameters.

• Config files- (.json) define model architecture, tokenizer, and generation settings.

• Tokenizer files convert text to tokens and back.

• Metadata (README.md) provides usage details.

Fig. 6 depicts the original model size (weights+ biases). You can see the bigger model like Mistral 7B has weights around ~ 15GB, while smaller models (Gemma -2- 2B) size is around ~5GB. So, one can select different set of models depending on the compute capacity and use-cases.

LLM Quantization

Model quantization is a critical technique in machine learning that reduces the precision of numerical representations. Like converting high-precision floating-point values (FP32, FP16) to lower-precision formats (INT8). Quantization significantly decreases model size, memory usage, and computational demands. This process facilitates the deployment of powerful language models like LLM’s on resource-constrained devices (such as laptop), improving efficiency and accessibility while typically incurring minimal performance degradation. I will write in detail about ML Quantization and techniques in a separate post.

There are different file formats for quantized LLM models. Fig. 7 shows a summarized table of some of these formats. The most famous quantized LLM format is GGUF (GPTQ/GGML Unified Format). It is a file format used for optimized LLM inference, particularly with CPU-based frameworks like llama.cpp, llamafile, and KoboldCpp.

Quantized LLM File Information and Usage

Figure 8 illustrates various quantized Mistral models, ranging from 8-bit to 2-bit, alongside a comparison with the original model. It's evident that as quantization increases, model weight decreases, making the model more suitable for resource-constrained devices. Notably, the 2-bit quantized Mistral 7B model is five times smaller than the original model. However, this size reduction comes with a performance trade-off, which we will discuss in the subsequent section.

Now let’s see what are these different quantization types. Fig. 9 presents a comparison table of different quantization types, detailing their descriptions and performance trade-offs. It highlights the trade-off between accuracy and speed, ranging from 2-bit (fastest but least accurate) to 8-bit (highest accuracy but slower). The table helps in selecting the appropriate quantization level based on optimization needs.

Understanding these trade-offs is crucial, but how do you decide which quantization type to use in practice? To help simplify this decision based on your specific hardware and use case, Fig. 10 provides a summarize guide for selecting the appropriate quantization type based on system specifications. It categorizes use cases from low-RAM (4GB or less) devices to high-performance setups (16GB+ RAM, strong GPUs or CPUs). The recommended quantization levels range from Q2_K for minimal resources to Q8_0 for the highest quality and accuracy.

LLM Serving

Okay, so we've talked about getting these LLMs running inference, which is basically asking them to do their thing. But how do we make that inference available to everyone who wants to use it? That's where LLM serving comes in, and it's a whole different ballgame we need to dive into.

Model serving has emerged as a critical component in the machine learning lifecycle, enabling organizations to transform trained models into valuable business applications. It is the broader infrastructure and system design (Fig. 11) that enables inference at scale and involves deploying the trained model as an API or service (via REST or gRPC APIs) that can handle multiple inference requests from users or applications. Model serving ensures that inference is efficient, scalable, and reliable. Organizations can implement model serving through several architectural patterns, each offering different trade-offs in terms of complexity, resource efficiency, and operational characteristics.

Let's break down how an LLM actually works in a system. Check out Figure 12. You will see the LLM at the heart of it all. To get it to do its thing, we use an inference library – that's the magic behind running the model. You can choose to run it on a CPU or, for a speed boost, a GPU. And to connect everything together, there's an API, acting as the bridge for users and other systems to interact with the LLM.

LLM Serving Frameworks for Different Scenarios

The landscape of LLM serving frameworks has expanded rapidly, with various solutions addressing different aspects of the serving challenge. These frameworks can be categorized based on their primary focus and capabilities, making them suitable for different deployment scenarios.

Lightweight Local LLM Hosting Frameworks

These frameworks are designed to run LLMs efficiently on consumer hardware (like CPU/GPU). They prioritize ease of use and minimal resource consumption, making LLMs accessible to a wider audience without requiring high-end servers. This allows for experimentation and development on personal devices.

Advanced LLM Inference Frameworks

For high-performance LLM hosting on GPUs and multi-node clusters. These frameworks are essential for applications requiring low latency and high throughput, particularly when dealing with large models and complex workloads. They leverage optimized techniques like tensor parallelism, KV Caching, pageAttention and distributed computing to maximize performance.

Local LLM Hosting with Quantization (Optimized for Low RAM & CPU):

These frameworks allow quantization (reducing model size) to run LLMs on low-end machines.

Multi-Model & Distributed LLM Hosting

For multi-GPU and multi-node distributed inference.

LLM UI Frameworks

Building user interfaces for LLMs used to be a real headache. You would have to cobble together a bunch of different tools, which wasn't ideal. Thankfully, we're seeing some great UI frameworks popping up that are specifically designed for LLMs. These frameworks make it easier to build intuitive and interactive interfaces, handling things like complex prompts, streaming responses, and even conversational memory.

Framework Usage Recommendations

In summary, the optimal framework for hosting your local LLM depends heavily on your specific needs and available resources (see Fig. 18). Whether you require a simple, local setup, advanced GPU inference, or a distributed, multi-GPU solution, the landscape offers diverse tools like Ollama, vLLM, and DeepSpeed. By carefully considering your use case and hardware, you can effectively deploy and leverage the power of large language models within your own environment.

Practical Usage

Now let’s see some methods to load these LLM models and Perform Inference-

Hugging face Transformer Library

The Hugging Face Transformers library is a powerful tool that makes working with large language models (LLMs) much easier. It's like a toolbox filled with pre-built models and tools, letting you quickly use and adapt cutting-edge AI. Whether you're interested in text generation, translation, or understanding language, Transformers simplifies the process. Lets see how it works

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# path to your locally downloaded Mistral 7B model
local_model_path = "/Users/piyush/Desktop/Codes/model/Mistral-7B-Instruct-v0.3"

# load the tokensizer from the local directory
tokenizer = AutoTokenizer.from_pretrained(local_model_path)

# check hardware cpu/gpu or mps for apple
device = "mps" if torch.backends.mps.is_available() else "cpu"

# load model 
model = AutoModelForCausalLM.from_pretrained(
    local_model_path,
    torch_dtype=torch.float16,  # Use float16 for lower memory usage
    device_map={"": device}             # Load on Apple Metal GPU if available
)

# Function to generate text
def chat(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=100)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Test the model
prompt = "Which part of Indian cosntitution is borrowed from the Ireland?"
response = chat(prompt)
print(response)

The output above displays the response generated by the Mistral 7B 16-bit model, using the Hugging Face Transformers library. You'll notice a sharp increase in memory usage (video) when the model is loaded. As mentioned before, it's important to choose models that match your hardware and intended use.

llama.cpp

Llama.cpp is a lightweight and efficient library designed to run LLMs on CPUs, even with limited resources. It excels at running models in GGUF format, making it ideal for devices with low RAM. This library allows users to experience the power of LLMs without requiring dedicated GPUs, broadening accessibility to these powerful AI tools. Let’s see how it works-

from llama_cpp import Llama

# laod the mistrl gguf model
llm = Llama(
    model_path="/Users/piyush/Desktop/Codes/model/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q8_0.gguf"
)

# define user and system prompts
messages = [
    {"role": "system", "content": "You are an expert assistant"},
    {"role": "user", "content": "Explain the concept of quantum computing in simple terms."}
]

# Generate response
response = llm.create_chat_completion(messages=messages)

# Extract and print the assistant's reply
generated_text = response["choices"][0]["message"]["content"].strip()
print("Assistant:", generated_text)

vllm and Ollama

If you're working on a Linux system, vLLM is an excellent option for efficient LLM inference on GPU. However, macOS support for vLLM is currently limited. As of today, achieving optimal performance on macOS typically requires building vLLM from source, which can be a more complex process.

Ollama is best to use for general audience with a one liner commad

ollama run mistral <or any other model>

Conclusion

I hope this overview has given you a clearer picture of LLMs, from their inner workings to practical deployment. We've covered a lot of ground, but this is just the beginning! In the next series of posts, we'll dive into building a real-world LLMOps framework. We'll walk through the process of creating an API and a chat UI, so you can interact with your LLM. Thanks for reading !!.

References

1. Sculley, David, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. "Hidden Technical Debt in Machine Learning Systems."¹ Advances in Neural Information Processing Systems 28 (2015).²

2. "MLOps for Better Machine Learning Deployment." Codilime Blog. Accessed. https://codilime.com/blog/mlops-for-better-machine-learning-deployment.

3. Remthix, S. D. "Discovering MLOps." Medium. Accessed. https://sdremthix.medium.com/discovering-mlops-a49ef3139696.

4. "What Is LLMOps?" Pluralsight Resources Blog. Accessed. https://www.pluralsight.com/resources/blog/ai-and-data/what-is-llmops.

5. "LLMOps." Databricks Glossary. Accessed. https://www.databricks.com/glossary/llmops.

6. "LLMOps vs. MLOps: Understanding the Differences." Iguazio Blog. Accessed. https://www.iguazio.com/blog/llmops-vs-mlops-understanding-the-differences/.

7. "LLMOps." lakeFS Blog. Accessed. https://lakefs.io/blog/llmops/.

8. https://www.tensorops.ai/post/what-are-quantized-llms

9. https://www.datacamp.com/tutorial/quantization-for-large-language-models

10. AI Engineering. O'Reilly. Accessed. https://www.oreilly.com/library/view/ai-engineering/9781098166298/.

11. "Transformers Documentation." Hugging Face. Accessed. https://huggingface.co/docs/transformers/en/index.

12. "vLLM CPU Installation on Apple Silicon." vLLM Documentation. Accessed. https://docs.vllm.ai/en/latest/getting_started/installation/cpu.html?device=apple.

13. "Ollama." Accessed. https://ollama.com/

14. ."koboldcpp." GitHub. Accessed. https://github.com/LostRuins/koboldcpp.

15. "LlamaIndex." Accessed. https://www.llamaindex.ai/

16. "LangChain." Accessed. https://www.langchain.com/

17. "Haystack." Deepset AI. Accessed. https://haystack.deepset.ai/

18. "LM Studio." Accessed. https://lmstudio.ai/

19. "Streamlit." Accessed. https://streamlit.io/

20. "llama.cpp." GitHub. Accessed. https://github.com/ggml-org/llama.cpp.

21. "exllama." GitHub. Accessed. https://github.com/turboderp/exllama.

22. "Open WebUI." Accessed. https://openwebui.com/

23. "OpenLLM." GitHub. Accessed. https://github.com/bentoml/OpenLLM.

    Subscribe now

    Thanks for reading Gradientor! This post is public so feel free to share it.

    Share

    Leave a comment

Abstract

The paper introduces VidCEP, a novel Complex Event Processing (CEP) framework designed for analysing video streams to detect spatiotemporal patterns. It addresses limitations in current CEP systems by using a graph-based representation of video data named Video Event Knowledge Graph (VEKG). VidCEP enables users to formulate expressive queries using a new Video Event Query Language (VEQL), facilitating the detection of events like object recognition and traffic flow monitoring. The framework then matches these queries against video content in near real-time, with demonstrated performance achieving a throughput of 70 frames per second. The system aims to bridge the gap between low-level video data and high-level semantic understanding, enabling more sophisticated video analytics. The research also evaluates the framework's performance with experiments that measure event representation time, accuracy, latency and overall throughput.

Reference

1. Yadav, Piyush, and Edward Curry. "Vidcep: Complex event processing framework to detect spatiotemporal patterns in video streams." In 2019 IEEE International conference on big data (big data), pp. 2513-2522. IEEE, 2019.

2. Google Gemini, NotebookLM

Part1: Understanding the Dirichlet Distribution: Basics

Part2: Hyperparameter Tuning of Neural Networks 👈 (You are here)

Introduction

This is part 2 of the Dirichlet series, focusing on its applications. In AI/ML (especially in NLP), the Dirichlet distribution is widely used in topic modeling. In coming weeks, I will be covering some interesting topic modeling concepts and their applications in LLMs. In this blog post, I will explore another important dimension that plays a significant role in training a neural network i.e. hyperparameter tuning. There is a ton of literature and tools available for selecting optimal hyperparameters. In this blogpost, I will be discussing from the perspective of applying the Dirichlet distribution to hyperparameter sampling. I'll also touch on some widely used hyperparameter tuning algorithms and compare them with the Dirichlet approach. Just a word of caution: the Dirichlet distribution is just one way to approach hyperparameter selection, and I'm showcasing its applicability within the wider context of AI/ML. Let's deep dive (or should I say, shallow dive? 🙂) into the world of neural network hyperparameter tuning from the perspective of Dirichlet lens.

Hyperparameters in Neural Network

Hyperparameters are settings or configuration parameters that define the structure of a neural network and influence the training process. You can think of these as set of knobs [4] that are set before the training and remain constant throughout the training phase. hyperparameters must be carefully selected to achieve optimal performance as impacts the training time, weights and biases (learned during the training) and many more... Below is a compherenshive flowchart and detailed table of different hyperparameters in a neural network.

Dirichlet Distribution in Hyperparameter Tuning of Neural Networks

Hyperparameter Tuning

Hyperparameter tuning is about selecting the best configuration (like learning rates, dropout rates, and activation functions) for training neural networks. The challenge is balancing exploration (trying diverse sety of configurations) and exploitation (focusing on promising/best/optimal configurations).

Dirichlet Role

As mentioned earlier, Dirichlet distribution can be useful when we want to generate probability distributions over multiple hyperparameter choices. This is where we utilize the role of alpha parameters (that we learned in earlier post). Instead of manually selecting hyperparameters, we used the Dirichlet sampling to efficiently explore diverse configurations by constraining certain values (like dropout, learning rates, or layer configurations) to sum to 1.

In a nutshell, dirichlet can help in -

• Generate diverse and controlled probabilities for hyperparameter values.

• Provides a controlled randomness

• Encourage sampling weighted configurations based on specific prior preferences for each parameter (like dropout or learning rates).

Code

Let’s understand this by code (git link)-

Step1- Import libs

# import libs (importing all in a single go)
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.animation import FuncAnimation
from scipy.stats import dirichlet
import tensorflow as tf
from tensorflow.keras import layers
import time
from itertools import product
import random

Step 2- Download Data and create a Neural Network Model

Let’s now load a toy dataset. MNIST (digits 0-9) [5] is one of standard dataset to test ML models performance. Also, lets create a toy/simple neural network (2 dense layers sequential model) using keras (use your own loved DNN framework).

# Load the MNIST dataset
def load_data():
    (X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
    X_train, X_test = X_train / 255.0, X_test / 255.0
    return (X_train, y_train), (X_test, y_test)

# Define the neural network model
def build_model(learning_rate, dropout_rate, hidden_size):
    model = tf.keras.Sequential([
        layers.Flatten(input_shape=(28, 28)), #  28 is image dimension of the MNIST datasgets
        layers.Dense(hidden_size, activation='relu'),
        layers.Dropout(dropout_rate),
        layers.Dense(10, activation='softmax')
    ])
    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
    model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    return model

Step3- Select list of hyperparameters

We need to select a sample hyperparameters for Dirichlet Distribution. For simplicity we have selected list of 3 most used hyper parameters.

• Dropout rate (alpha =3)

• Hidden Layer Size (alpha =2)

• Learning Rate (alpha =1)

# Generate hyperparameters using Dirichlet distribution
def dirichlet_sampling():
    alpha = [3, 2, 1] #  seeting a random alpha
    samples = dirichlet.rvs(alpha, size=50)#  setting a sample size of  50
    learning_rates = samples[:, 0] * 0.01
    dropout_rates = samples[:, 1] * 0.5
    hidden_layer_sizes = (samples[:, 2] * 100).astype(int) #  setting int as hidden layers are in int
    return list(zip(learning_rates, dropout_rates, hidden_layer_sizes))

Step4- Evaluate set of hyperparameter configurations

Now lets evaluate the hyperparameter configuration. We will pass the hyperparameters value, train and test datastet and build the model. For sake of simplicity we will be evaluating the model performance on two parameters-

• Accuracy

• Training Time

The fucntion will loop in all the hyperparams value and store the accuracy and training time as a performance metric result. Basically the set of hyperparams (learning rate, dropout, hidden layer size) that gives higher accuracy with less training time is best.

Note- To avoid computational bottleneck and test the concept we have set the epoch at 1. In real world setting it will be higher.

# Evaluate a list of hyperparameter configurations
def evaluate_hyperparameters(hyperparameters, X_train, y_train, X_test, y_test):
    results = []
    for lr, dr, hs in hyperparameters:
        model = build_model(lr, dr, hs)
        start_time = time.time()
        history = model.fit(X_train, y_train, epochs=1, validation_data=(X_test, y_test), verbose=0)
        training_time = time.time() - start_time
        accuracy = history.history['val_accuracy'][-1]
        results.append((accuracy, training_time, lr, dr, hs))
    return results

Pareto Optimal Points and Curve

Before executing the code lets also write a small code to get the pareto points from the hyperparams set. Pareto Optimality is all about finding the "sweet spots" where you can't improve one thing without making the other worse. Lets say we are tuning two hyperparameters for your neural network:

• Learning Rate

• Number of Layers

And we want to find the "best" combination of these two on basis of

• Accuracy: How well the network performs.

• Training Time: How long it takes to train the neural network

A Pareto point is a specific combination of learning rate and number of layers where you can't improve both accuracy and training time at the same time.

Pareto Curve is the set/line joining all pareto points.

Why Pareto Analysis is Useful

The Pareto curve helps us to visualize the trade-offs. We can then choose a point on the curve that best suits our needs. Maybe someone need a model that trains quickly, so he will choose a point with lower accuracy but short training time. Or perhaps for someone accuracy is paramount, and s/he is willing to wait longer for training. The Pareto curve makes these decisions clearer.

Note: For the sake of blog lenght I am not writing the pareto code here. You can directly look for get_pareto_points and animate_pareto functions in the code repo.

Step 5- Test the dirichlet sampling method

Now lets run the setup to test the performance of different hyperparameters.

# Load the dataset
(X_train, y_train), (X_test, y_test) = load_data()

# generate hyper parameter configurations
dirichlet_params = dirichlet_sampling()

# evaluate the configurations
performance_results_dirichlet = evaluate_hyperparameters(dirichlet_params, X_train, y_train, X_test, y_test)

# get the pareto points
animate_pareto(performance_results_dirichlet)

In the above results, we can see five pareto points in 50 samples of hyper parameters. Its now upto the user to select optimal hyper parameter samples as per their requirements like., for higher accuracy (0.959) they can select the first sample (fig 3) and for less training time the 3rd sample (fig3) is preferred.

Benchmarking with other SOTA hyperparameter tuning algorithms

Finally, I will discuss a brief benchmark analysis with standard hyperparameter tuning algorithms. While this article focuses on the applicability of Dirichlet distributions rather than standard hyperparameter tuning algorithms, it would be beneficial to briefly review these algorithms (Fig5) and commonly used tools (Fig6).

Grid Search and Random Search are two famous vanilla hyper prams tuning algos. Lets see them in action.

Grid Search

Grid search is an exhaustive algorithm that spans all the combinations, to find the best points in the sampel set. Grid search tries every single combination of hyperparameter values (Fig7- left) within the ranges you specify, trains a model for each combination, and then picks the combination that gives the best results.

Example- Suppose hyperparams (A, B) are A = {0.1,0.2}, B= {0.3, 0.4}. Then grid search will create all combinations i.e. [ {0.1,0.3}, {0.1,0.4}, {0.2,0.3}, {0.2,0.4}]

Cons- Super slow, computanially intensive.

# Generate hyperparameters using Grid Search
def grid_search():
    grid_learning_rates = [0.001, 0.005, 0.01]
    grid_dropout_rates = [0.1, 0.3, 0.5]
    grid_hidden_layer_sizes = [10, 50, 100]
    return list(product(grid_learning_rates, grid_dropout_rates, grid_hidden_layer_sizes))

Random Search

Random search is a method for finding the best settings (hyperparameters) for a machine learning model, but instead of trying every possible combination like grid search, it picks combinations randomly. So fro above example, random search will be a subset selection like [{0.1,0.4}, {0.2,0.3}]

# Generate hyperparameters using Random Search
def random_search():
    random_learning_rates = np.random.uniform(0.001, 0.01, 10)
    random_dropout_rates = np.random.uniform(0.0, 0.5, 10)
    random_hidden_layer_sizes = np.random.randint(1, 100, 10)
    return list(zip(random_learning_rates, random_dropout_rates, random_hidden_layer_sizes))

Lets also create a 3d scatter plot to visualize the hyperparams features.

# Plot 3D hyperparameter configurations
def plot_3d_hyperparameters(results_by_method, methods):
    fig_3d = plt.figure()
    ax_3d = fig_3d.add_subplot(111, projection='3d')
    for i, results in enumerate(results_by_method):
        lrs = [result[2] for result in results]
        drs = [result[3] for result in results]
        hls = [result[4] for result in results]
        ax_3d.scatter(lrs, drs, hls, label=f'{methods[i]}', marker='o')
    ax_3d.set_xlabel('Learning Rate')
    ax_3d.set_ylabel('Dropout Rate')
    ax_3d.set_zlabel('Hidden Layer Size')
    plt.title('3D Plot of Hyperparameter Configurations')
    plt.legend()
    plt.show()

Now test and benchmark the two methods with dirichlet.

# generate hyper params config
grid_params = grid_search()
random_params = random_search()

# perform grid search
performance_results_grid = evaluate_hyperparameters(grid_params, X_train, y_train, X_test, y_test)

# perform random search
performance_results_random = evaluate_hyperparameters(random_params, X_train, y_train, X_test, y_test)

# Plot results
methods = ['Dirichlet', 'Grid Search', 'Random Search']
all_results = [performance_results_dirichlet, performance_results_grid, performance_results_random]

# add 3d visualiuzation
plot_3d_hyperparameters(all_results, methods)

# display best hyperparameter samples
print("Best Hyperparameter Samples by Method:")
for method, results in zip(methods, all_results):
  best_sample = max(results, key=lambda x: x[0])
  acc, time_taken, lr, dr, hs = best_sample
  print(f"{method} -> Accuracy: {acc:.4f}, Time: {time_taken:.2f}s, LR: {lr:.5f}, Dropout: {dr:.2f}, Hidden Size: {hs}"

Conclusion

Alright, it’s a wrap! I have covered the basics of neural network hyperparameters and some common tuning methods. Remember, this article took a closer look at using Dirichlet distributions for AI/ML applications, with a quick benchmark of grid and random search for comparison. These benchmark results are specific to my setup – there's no single "best" tuning method, it all depends on your use case and experiment settings. Hope you found this helpful and enjoyed the read! 😊

References

1. Kadhim, Zahraa Saddi, Hasanen S. Abdullah, and Khalil Ibrahim Ghathwan. "Artificial Neural Network Hyperparameters Optimization: A Survey." International Journal of Online & Biomedical Engineering 18, no. 15 (2022).

2. Yu, Tong, and Hong Zhu. "Hyper-parameter optimization: A review of algorithms and applications." arXiv preprint arXiv:2003.05689 (2020).

3. Liao, Lizhi, Heng Li, Weiyi Shang, and Lei Ma. "An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks." ACM Transactions on Software Engineering and Methodology (TOSEM) 31, no. 3 (2022): 1-40.

4. Yadav, Piyush, Dhaval Salwala, and Edward Curry. "Vid-win: Fast video event matching with query-aware windowing at the edge for the internet of multimedia things." IEEE Internet of Things Journal 8, no. 13 (2021): 10367-10389.

5. MNIST, http://yann.lecun.com/exdb/

6. Neptune Blog: Best Tools for Model Tuning and Hyperparameter Optimization https://neptune.ai/blog/best-tools-for-model-tuning-and-hyperparameter-optimization

7. Github- Hyper-parameter tuning library https://github.com/balavenkatesh3322/hyperparameter_tuning

8. Hyperparameter tuning. Grid search and random search https://www.yourdatateacher.com/2021/05/19/hyperparameter-tuning-grid-search-and-random-search/

Leave a comment

Part1: Understanding the Dirichlet Distribution: Basics 👈 (You are here)

Part2: Hyperparameter Tuning of Neural Networks

Part2: Understanding the Dirichlet Distribution: AI/ML Applicationns (coming soon)

Introduction

I have recently been reading about a new topic on Theory of Mind that basically originates from Bayesian statistics. While exploring how to define the beliefs of different agents, I encountered the Dirichlet distribution and thought it would be worthwhile to share its relevance to the AI/ML field. The Dirichlet distribution is a powerful and versatile probability distribution with applications in various fields, including machine learning, statistics, natural language processing, and image analysis. This blog post provides a deep dive into the Dirichlet distribution and some of its applied use cases in different AI domains.

Drichlet Distribution

Dirichlet distribution might sound complicated, but it's actually a useful concept when we want to describe how probabilities are divided among different categories. It is a probability distribution over multiple categories. Think of it as helping decide how much weight or probability each category should get.

For Example-

Example1- Suppose you want to divide a fixed amount of money (1 USD, it can be any number but later normalized to 1) among friends (lets say 3). The Dirichlet distribution helps us figure out the many possible ways this money (or probability) can be shared.

Example 2- Similarly, let’s say you want to predict how likely people are to vote for three candidates. As shown in above figure, in scenario 1, you believe A (50% votes) and C (30%) are more popular than B (20%).

Mathematical Explanation

The probability density function (PDF) for the Dirichlet distribution is defined as:

In above formula-

• x₁​,x₂​,...,x_K​ are the proportions (which must sum to 1).

• α₁,α₂,…,α_K are the parameters for each category.

• Γ is the Gamma function .

Understanding alpha (α) parameters

The Dirichlet distribution is controlled by a set of numbers called parameters, written as α=[α₁,α₂,…,α_K]. These parameters decide how balanced or extreme the distribution of probabilities will be.

Uniform Distribution

α =1: All combinations are equally likely.

When all α_i=1, the PDF simplifies, making all combinations of probabilities equally likely. Lets understand this by code-

# Import libs
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import dirichlet

# Plot the samples on a 2D simplex (triangular plot)
# For more on triangular/simplex and barycentric plots-
# https://en.wikipedia.org/wiki/Ternary_plot

def plot_simplex(samples):
    fig = plt.figure(figsize=(8, 6))
    ax = fig.add_subplot(111)

    # Convert to barycentric coordinates for plotting
    x = samples[:, 0] + 0.5 * samples[:, 1]
    y = np.sqrt(3) / 2 * samples[:, 1]

    ax.scatter(x, y, alpha=0.1, edgecolor='k')
    ax.set_title('Samples from Dirichlet Distribution')
    plt.show()

# Parameters for Dirichlet distribution
alpha = [1, 1, 1]

# Generate samples
samples_uniform = dirichlet.rvs(alpha, size=5000)

plot_simplex(samples_uniform)

All combinations are equally likely. This is called a uniform distribution, meaning each possible probability scenario (like [0.3,.0.,0.4] or [0.5,0.2, 0.3] ) is equally likely. Think of it as having no prior bias toward any specific way of distributing probabilities. Basically you have no preference for how the votes are split. Any division is equally acceptable. Thats the plot shows a dense scatter across the entire simplex (triangle). There is no bias toward any specific outcome.

Balanced Distribution

When α >1: All combinations are equally likely.

Larger α​ values (all equal) favor more balanced distributions near the center of the simplex. For Example: Think of distributing votes among candidates where you expect relatively even support among them, though not perfectly strict

# Parameters for Dirichlet distribution
alpha = [6, 6, 6]

# Generate samples
samples_balanced = dirichlet.rvs(alpha, size=5000)

plot_simplex(samples_balanced)

Probabilities are more evenly distributed and cluster around the center of the triangle ([0.33,0.33,0.33]). The samples favor balanced scenarios where categories share probabilities evenly.

Skewed Distribution

Smaller values of α ​<1 favor extreme outcomes. One category tends to dominate, while others shrink toward zero. Example: In elections, perhaps one candidate might unexpectedly take the majority of votes.

# Parameters for Dirichlet distribution
alpha = [0.8, 0.3, 0.3]

# Generate samples
samples_skewed = dirichlet.rvs(alpha, size=5000)

plot_simplex(samples_skewed)

The distribution favors extreme points. You can see more samples near the vertices of the triangle, indicating that one category dominates while the others receive very little.

Why this is happening???

Dirichlet samples are derived by generating Gamma-distributed random variables and normalizing them.

Basically-

• Larger α values produce Gamma samples with less variance and values clustered near the center.

• Smaller α values result in more extreme values.

  from scipy.stats import gamma
  import numpy as np
  import matplotlib.pyplot as plt
  
  def plot_gamma_samples(alpha_value, size=5000):
      # Generate Gamma samples for three categories
      y1 = gamma.rvs(alpha_value, size=size)
      y2 = gamma.rvs(alpha_value, size=size)
      y3 = gamma.rvs(alpha_value, size=size)
      
      # Normalize them to sum to 1 (Dirichlet transformation )
      total = y1 + y2 + y3
      x1 = y1 / total
      x2 = y2 / total
      x3 = y3 / total
      
      # Plot the Gamma samples before normalization
      plt.figure(figsize=(8, 4))
      plt.hist(y1, bins=50, alpha=0.6, label=f'Gamma samples (alpha={alpha_value})')
      plt.legend()
      plt.title(f'Gamma Distribution for alpha={alpha_value}')
      plt.show()
  
      # Plot normalized Dirichlet samples
      plt.figure(figsize=(6, 3))
      plt.scatter(x1, x2, alpha=0.1, edgecolor='k')
      plt.title(f'Normalized Dirichlet Samples (alpha={alpha_value})')
      plt.show()
  
  # Visualizing Gamma and Dirichlet samples for alpha values
  plot_gamma_samples(1)  # Uniform
  plot_gamma_samples(6)  # Balanced
  plot_gamma_samples(0.5)  # Skewed
  

  

Gamma Distribution for α=1 (Uniform)

The Gamma samples are broadly distributed across different values.After normalization, the points scatter uniformly across the entire simplex.

Gamma Distribution for α=6 (Balanced)

The Gamma samples are tightly clustered, leading to balanced and less variable probabilities. After normalization, the points tend to cluster near the center.

Gamma Distribution for α=0.5 (Skewed)

The Gamma samples are concentrated near small values, often close to zero. After normalization, the points scatter near the simplex vertices, indicating one dominant category.

Gamma distribution defines the "weight" each category gets. By dividing by the total weight, itsd creates probability values for each category.

 α>1: Produces more balanced weights, favoring smoother, less extreme probabilities.
 α<1: Produces spiky, peaky weights, where some categories get much more weight than others.

In the next series, we will deep dive into different practical AI applications where Drichlet is used.

Leave a comment

Share

Leave a comment

Hello World