A Brief History of the Database: From Punch Cards to Vector Embeddings

This was initially supposed to be a simple tutorial on how to get started with Turso: a SQLite compatible database built for the cloud in the age of AI. In the process, I got sidetracked with the history of database development. I believe it’s worth revisiting the fundamentals to better understand some of the recent developments in the age of AI.

The Origin of the Database

A useful definition of a database is: an organized collection of data that’s easy to search and update.

Until the 1950s, paper and punch cards were the primary method of storing and managing data. Herman Hollerith built a famous punch card tabulating machine for the 1890s US census (his company later became part of IBM). Businesses relied on these cards for decades, and many kept using them well into the 1970s.

In the early 1950s, the UNIVAC I computer introduced magnetic tapes. One reel could now hold as much as a thousand cards. But you still had to read it in order from start to finish.

Then in 1956, IBM’s RAMAC brought magnetic disks and random access. This meant the computer could go straight to any record and this was a big deal for speed.

The 1960s saw the emergence of removable disk packs like the IBM 2311 in the photo below, and disks soon became the normal place to keep data.

Left: IBM 350 RAMAC (1956), photo by vnunet.com, CC BY-SA 2.5. Right: IBM 2311 (1960s), photo by Mikaël Restoux, CC BY 2.5. Both via Wikimedia Commons.

Once disks were common, the development of database software followed. Charles Bachman’s Integrated Data Store, developed at General Electric, came out in 1963. It kept manufacturing data on disks and shared it across programs. IBM developed the Information Management System (IMS) late in the 1960s, partly to track the millions of parts of the Apollo program.

The Relational Database Emerges

These early systems organized data as rigid trees, with every record having exactly one parent.

To find something, you started at the top and walked down a fixed path. Questions that followed the path were fast to answer, such as “What did Jane order?” Questions that cut across the paths, like “who ordered a lamp?” were painful. You would have to walk every customer’s tree to see if they ordered a lamp. Updating the structure also often meant rewriting the programs that used it.

In 1970, Edgar Codd at IBM proposed a new way of storing data: the relational model (the most famous model to this day). Instead of storing data as a tree of paths, he proposed to store it as plain tables, much like a spreadsheet. Each group of data would get its own table, and tables would know about each other using IDs.

(Relational simply means tables relating to one another through shared IDs).

For example, customer_id 1 in the Orders table points to Jane in the Customers table.

Then you simply tell the database what you want and the database works out the fastest way to get there. For example, “who bought a lamp?” becomes:

SELECT name FROM customers
JOIN orders ON orders.customer_id = customers.id
WHERE item = 'Lamp';

This produces Jane and Omar.

The same tables can now answer questions from any direction, and tables can be joined without rewriting the underlying programs that use them. That language later became known as Structured Query Language (SQL).

The World Wide Web Explodes

The first software databases were commercial, offered by companies like Oracle, IBM, Informix and Sybase. Then in the 90s, the World Wide Web exploded into popularity. Suddenly, there was a dire need for another type of database software, a free and open source one, which the millions of websites that were now live on the web could use - to store information about their millions of users and products.

MySQL, released in 1995, was one response to this demand. It was a relational database created and open-sourced by Michael Widenius and David Axmark at a small Swedish company. It became the “M” in the LAMP (Linux, Apache, MySQL, PHP) stack that powered much of the early web. Its benefits were that it was fast and lean.

MySQL operated under a dual license. It was free under an open source license, but if companies wanted to ship it inside their own closed products, they had to pay for a commercial license. MySQL later got acquired by Oracle. Its licensing restrictions and being absorbed by a large company concerned developers. These concerns pushed many toward an older open source database that had been quietly maturing for years: Postgres.

Postgres: Free and Open-Source for Everyone

Here’s a quick recap of how Postgres came to be.

In the 1970s, based on Edgar Codd’s relational ideas, Michael Stonebraker (UC Berkeley) built one of the first relational databases and called it Ingres (Stonebraker later won the Turing Award, the top prize in computer science, partly for this work).

In 1986, Stonebraker started a new project to fix Ingres’s limits, and named it Postgres, meaning “post Ingres.” It had its own query language instead of SQL. From 1994 to 1995, two Berkeley grad students, Andrew Yu and Jolly Chen, swapped in SQL and released it for free as Postgres95.

In 1996, it was renamed PostgreSQL to show it now used SQL. A volunteer community took over development, which is still how it’s run today. Postgres has also found a second life with AI, but more on that later.

Postgres earned its reputation by being strict. It rejected invalid data instead of quietly saving it, made related changes happen all together or not at all, and could recover cleanly after a crash. It was also built so anyone could add new features on top of it, which is why it kept finding new uses over the years.

Today Postgres is a default choice for many new apps, and companies like Neon and Supabase are built entirely on it. Other relational databases like Oracle, MySQL, and Microsoft SQL Server continue to be used, largely because enterprises like banks, governments and airlines have used them for decades and moving away is costly.

Databases Go Small Too: SQLite

In 2000, D. Richard Hipp was working on software for the US Navy. The software’s database required a running server, and if the server went down, then the software failed too. He wanted an alternative solution, a database that the software could read and write to directly, with no server required. So he created SQLite, a full relational database that lives inside a single file in an application, with no server required to run it.

SQLite took off. At the same time that Postgres was optimizing for how millions of users can share the same data on a server, SQLite was optimizing for how millions of app instances could store and access their own data.

Before SQLite, apps on a device usually saved data to their own custom file formats. Developers had to write code to save, load and search that data themselves, and it broke easily. SQLite standardized it. It gave every app a real database inside a single file, with tables, SQL and safe writes, and without having to install or run a server. Its creator famously said SQLite doesn’t compete with Postgres, it competes with saving to a plain file.

Today, SQLite is the most widely used database in the world. It runs inside every iPhone and Android device, many web browsers, Macs, and countless other apps. It’s also in the public domain, so anyone can use it for anything for free.

SQLite is worth resurfacing because its design fits the AI age. Each database is just a file, so you can give every agent its own memory, create thousands of them cheaply, copy one before letting an agent experiment, and run it anywhere without a server. Projects like Turso are building on this idea.

The Rise of Vector Search

Traditional databases are great at exact matches, like finding every order from a customer or every product under $50. But AI apps deal with meaning. To capture meaning, an AI model turns a piece of text, an image or a sound into an embedding (which is a long list of numbers). Things with similar meanings end up with similar numbers. So the embeddings for ‘cheap flights’ and ‘budget airfare’ sit close together even though they share no words.

Storing these lists of numbers, called vectors, and quickly finding the ones closest to a question is called similarity search. It’s how a chatbot finds the most relevant parts of your documents before it answers, and it’s why a new category called vector databases appeared.

Postgres didn’t need to be replaced to handle this. In 2021, Andrew Kane released pgvector, an open source extension that lets Postgres store embeddings and run similarity search. Because the embeddings live right next to your regular data, you can combine both in one query, like finding the most relevant products that are also in stock and under $50. So instead of adding a separate vector database, many teams building AI apps just use the Postgres they already have.

Conclusion

This is far from the whole story. In the 2000s and 2010s, the explosion of huge websites gave rise to NoSQL databases like MongoDB and Redis, which traded strict tables for flexibility and speed. The cloud turned databases into services you rent instead of servers you run. Analytics databases like Snowflake and ClickHouse were built to crunch massive amounts of data, and distributed databases like Google Spanner spread a single database across the world. Each of these deserves its own article.

But the pattern is the same one that started with punch cards: every time the way we build software changes, the database changes with it. Tape gave way to disks, trees gave way to tables, and big shared servers were joined by tiny databases living inside a single file. Today, that change is AI. Apps now need to search by meaning, and AI agents need somewhere to store what they know and remember what they did. The old ideas are holding up surprisingly well. Postgres learned to store embeddings, and SQLite’s idea of a small, simple database that lives right next to the code fits agents well. Fifty years after Codd’s paper, we’re still building on tables.