I live in a country where the selection of available books, especially in English, is very limited. Buying online from foreign markets comes with a long list of administrative hurdles and limits.
If it were not for Anna's Archive and Z-Library, I would've never been able to read the books that shaped who I am today, or keep my passion for learning alive.
Thanks, AA and ZLib! (Also, thank you to the authors whose books and knowledge I consumed without being able to pay them back.)
> Knowledge should be free. It was never created in a vacuum.
This is a common perspective on HN, but it's so jarring. Someone violates an open-source license and we grab our pitchforks. Someone pirates books and it's fine - really, the authors should be thanking us.
Good books are incredibly challenging to write, more so than good software. It's not like you grab Harry Potter and say "I'm just gonna change character names and rephrase some of the text". Most authors recognize that not everyone can afford books and then contend that some amount of sharing is healthy, no different from borrowing books from a local library. If you ask nicely, they will probably send you a PDF for free. But the scale of online book piracy is absolutely staggering and demoralizing, and most of it has nothing to do with taking any serious moral stance. It's just "lol, why pay when you can download for free".
If this is not bad faith argument, then I don't what is. When someone is violating an OSS licence, they are doing it for commercial gains and monetary profit. Nobody is angry at someone using FOSS software for himself with no money getting involved.
As opposed to that, books, movies are pirated for personal consumption. Not monetary gains. If someone bought a $30 book, and then ran a BaaS with millions of VC money in his pocket, people in HN would be angry at him, too.
If knowledge is to be free, that means there should be no restrictions on how it is used. Even an open source license misses the point, because the implication is still that one person can dictate how another person can make use of knowledge. It’s still premised on the same dystopian view that a person can own an idea.
There’s also the argument that copyright has been extended to the point of absurdity.
I respect copyright, but I can’t respect a lockup period that can push to 130 years or more. For example: if JK Rowling is alive in 4 years the first Harry Potter book will have a valid copyright extending from the 20th, 21st and into the 22nd century. Is it really defensible to say that your great, great (great?) grandchildren should benefit from a government mandated monopoly on your work?
JK Rowling is the exception. The duration is one thing, but so are her proceeds. Most authors are never going to see that much money, if any, from writing. Maybe copyright would expire once the original author has made enough money to pay for lavish living expenses for themselves for the rest of their life, inflation adjusted. So their family wouldn't automatically be taken care of, but they are.
Kind of weird too think about it that way, but food for thought.
Sure, but for the lower volume authors the royalties at year 20 are effectively 0. Most books don’t even sell out their first printing. When you are shredding unsold copies of books, defending copyright is a pure money grab.
100+ year copyrights only help the descendants of the people that have already become extraordinarily wealthy. Cutting off at 2 decades would have close to 0 effect on the huge majority of creatives, while benefitting society immensely. JK Rowling would still be a billionaire, and small authors would be fine too.
> Someone violates an open-source license and we grab our pitchforks.
If you look back through the annals of Free Software, one often encounters the claim that the GPL was a way to use copyright against itself, and if there were no more copyright, there would be little need for these licenses.
The issue with copyright law is that it is all or nothing. Rights to a work are either tightly held by an author/publisher, and even downloading a small excerpt can get you in trouble, or it is fully public domain and open for any and all use.
There needs to be a middle ground, such as: after 15 years of publication any private individual can access and read the work for free, but the rightsholder still controls commercial sales, merchandising, licensing, character rights, movie rights etc.
Taken out of context, you're right. But the parent comment couldn't buy these books even if they wanted to. I'd say there's a consensus that the primary motivation for piracy is hurdles to access having nothing to do with payment.
> Knowledge should be free. It was never created in a vacuum. It belongs to us all
Imagine you're a professional writer and it's your main source of income. How would you feel if someone said this to you? Would you still want to write books?
Things have changed a lot since the late twentieth century. The kind of people you imagine, who can live full-time off writing, are responsible for a vanishingly small amount of the books that appear today. Piracy has little to do with it; this is primarily due to the fierce competition from other books, the glut of content available today, but especially from mobile phones as fewer and fewer people read books. Even for those who make appreciable income off books, the books are nevertheless usually a side gig alongside other hustles.
Imagine that terrible time before Copyright existed, and there was no motivation for anyone to make art, literature, or music.
"What an inestimable advantage it would be, if, in every branch of literature, there existed only a few but excellent books! This can never come to pass so long as money is to be made by writing." --Arthur Schopenhauer
This is such a ridiculous statement. The people who spent their time building up this body of work deserve to be compensated. Take whatever job you do and imagine people confidently stating that you should work for free.
But do their grandchildren deserve to live off the proceeds for the rest of their lives as well? Say I'm a carpenter and I make chairs. How many chairs do I need to make before I get to retire? If I make a really really good one and get it put it in the right place, one should be enough. Just sit back and collect $1 for every time that someone sits in it. I don't have to make the chair particular good or comfortable, just get it into the right place where people will pay. And then I don't have to work for the rest of my life. Nor do my children. Or their children either. Framing the question at the extreme, that one should be expected to work for free, is just as absurd as framing it as some people should just never have to work at all, ever. No one put me in charge, but I believe people need to do work of some sort. Who gets to decide what counts as work or not isn't for me to decide though, so the system we've got is just this whole unorganized unplanned economy.
Everything you’ve ever created, and ever will create, both digital and physical, do you mind if I grant myself a non-exclusive perpetual license to copy / manufacture and redistribute in unlimited quantities in any and all markets for all time?
Look, fair enough from your perspective. But a lot of those books probably wouldn't exist if the author couldn't make some money from their work.
I can't find the post but years ago on Reddit an author posted stats showing when her book turned up pirates online, real sales for it collapsed.
Because of this I make a point of buying books, programming books especially. Yes I download pdfs, I use them as previews. This has led to buying way more than I would have.
Anyway, I appreciate this doesn't apply if you live somewhere that these books can't be purchased. But everyone praising these sorts of sites tends to look at them from only a positive perspective.
I live in Iran and the administrative hurdles the op was mentioned are not an issue here because you just can't even buy intwrnationally to begin with so there is no hurdle you might need to circumvent. The few English books I have are largely illegal reprints of a pirated version or some old ancient printed version that have somehow gotten imported (no clue how or is there an actual legal way)
I remember opening "thinking fast and slow" and noticing the weird paging. After checking the official version's page count and seeing how the version in my hand doesn't match, my best guess was that someone had printed a pirated epub version.
I rambled so much that I forgot to say what I laid all that introduction for.
I'm not part of the market for these products. I don't have access to them nor even if I buy some imported (probably illegally and by single persons) or printed version, am I going to benefit them since I'm disconnected monetarily from yhe author.
Me reading pirated versions of these books has no negative effects on the earnings of the authors.
> But a lot of those books probably wouldn't exist if the author couldn't make some money from their work.
I think that's at least a bit debatable. People thought that about (normal) libraries back in the day, but it ended up having the opposite effect.
Not to mention out of print books or academic books which is a big usage of sites like these, since lots of people prefer physical books and only reach for pdfs as a last resort.
Libraries spend like $2B / year buying books https://www.imls.gov/sites/default/files/2021-08/fy19-pls-re..., which is like 10% of the total book market. So even if no one ever bought a book because they first encountered the book, author, or genre in the library that's already a signficant difference
I think I agree, the FAR bigger impact on my book's sales was Google search deciding not to surface it in search results. Presence on pirate websites had no effect, and eventually I switched to the PDF as "pay what you want."
Can you imagine if we didn’t have libraries and someone tried to create them today? From publishers to right wingers, they would be painted as communist plots to destroy creativity.
The Internet Archive tried, at great cost and peril, to defend its ability to lend books as an online library due to format shift (physical books get first sale doctrine, ebooks are licensed, you cannot own them), and were told no by the system, so “pirating” it is until copyright changes and becomes more reasonable. Disk is cheap, and the Internet global. Global distributed storage system durability and availability is the path to success until laws change imho.
(Archiving culture alone is not the same as also enabling universal access to the culture and knowledge one is acting as custodian for and serving to global citizens)
I'm glad your username specifies your location. My biggest pet peeve online these days is someone telling a story about "my country" but never specifying which country that is.
We sell ebooks in every country, including Tunisia - https://www.ebooks.com/en-tn/. I understand that the price of books is sometimes prohibitive, but it's largely outside our control.
Thanks, but I don't use e-readers as they are not available here.
I've been using MoonReader for many years now and settled on pretty good parameters that make the reading experience very comfortable on both my phone and my tablet.
On my Kindle I use KOReader Zlibrary plugin which allows you to download books from within the reader. It's more convenient then any send-to-kindle workflow.
I’ve noticed that people today often bristle at any suggestion that one connect a device to a phone or computer with a cable – on Reddit, one will often get downvoted for this. Apparently, a lot of younger people are hardly aware this is possible and it strikes them as overly complicated or for old people. People want to wirelessly transfer stuff, and what the OP linked to is a popular way to do that with Kobo.
I agree that a USB cable is the most practical option. However, the aforementioned site is useful in a specific scenario: if your Kobo is very old, macOS won't recognize it.
https://SourceLibrary.org has about 16,000 rare books translated — most for the first time. 50,000 books archived (will be translated when we have $$ for it). More tokens than English Wikipedia and about .75 petabytes.
Not sure if we will qualify for a bounty, but happy to share! Btw, we are looking for funding from small or large donors who want to help us translate the Renaissance…
I can't quickly tell what all you have archived^, but I have some friends who are academic historians who might be interested in certain categories of work (and could help verify some esoteric languages) - is it possible to search by region or language?
Have you reached out to any types of historians WRT the project? It seems like some PhD students might be able to find some projects in this work etc
Yes, this is designed with historians and librarians from the Embassy of the Free Mind (https://embassyofthefreemind.com) in Amsterdam, stewards of the collection of the Biblioteca Philosophica Hermetica
Please share with historian friends. I’m not great at socials or fundraising but this was really designed to support humanists. It can give DOIs for the versions of the translated books, which means they can be quoted and cited in academic papers.
Tip: Try it in Claude or Claude code (even better)! Just point it towards the source library. It can find quotes and evidence on any topic of interest. Or try the librarian — our source-grounded research agent https://sourcelibrary.org/librarian
Interesting site. I picked a random topic to listen to — flying chariots or something like that — and the conversation of one person talking and the other whispering was definitely not to my preference. I’ll have to take another look when I have more time.
I don't see raw token counts, just a list of steps and page counts. For example, what is the rough average token count per page in the ocr and in the translation steps for a Greek book?
I have seen Gemini costs change quite a bit when processing very similar books from the same series lately, mainly because thinking tokens have increased about 5x. Has that has happened to you as well?
Edit: for ocr I am using about 15k-25k tokens per page, but I have a complex prompt.
How do you handle the more densely written pages in script ? I did a very similar exercise OCRing works from this exact collection, but I stuck with the English books for the first pass.
Anna’s came clutch for me yesterday. I spent a few days trying to find a zip file of a CD that came with an old book from early 2000s on programming. One of those Thomson Publishing slap jobs that I actually enjoyed. I checked used copies all of them said does not come with CD. I tried googling around, nothing. LLMs couldn’t find it. ChatGPT kept saying it is on the archive (no it isn’t you useless piece of shit). Anyway, on a whim I went to AA, lo and behold, zip files for both first and second edition. Godsend.
I wonder how long it will be before they offer bounties for internet scrapes.
Cloudflare captchas have made the internet unusable for me, and I'm sure it will only get worse over time. I'd much rather just browse (or even torrent) a copy of archive.is or similar. The latter would be much better for privacy, and hey, I run ad blockers anyway.
Someone on your network is likely playing one of the games that are monetized by bright data proxies, it was a thread on here a few days ago. It could be your smart TV. If you find the culprit and remove it there's a decent chance your ip reputation will improve enough to not see those captchas
I think the main source may be in Russia; or that was with libgen.
But I could be wrong.
I am more surprised to see that there are so few alternatives to it. Or perhaps I am unaware of them but after Facebook and co declared war on libgen, and libgen going down, there were surprisingly few alternatives. Anna was one of the few. I still don't know what happened with libgen, but since the attack it really is kind of semi-gone.
Libgen and similar are more alive than ever with an extended botnet growing weekly. The "googlers" indexed framework is shrinking everyday, so users wont find it in those search engines easily, also it is hard to keep up with a good storage considering price trend last 5 years so the botnet and torrents are some kind of solution I guess. (We for instance are considering to use the old taping system, cause is at least a viable alternative.
I wonder how hard Google could press an estate. For a living person the main consequences would be criminal, not civil, but you can't go after a dead person criminally. Civilly it's not really clear to me that Google would have been harmed enough to create noteworthy damages.
The book publishers might actually be the bigger problem. They'd have civil copyright infringement claims with giant statutory damages.
It’s the “ Copy data into extra large capacity micro sdcard” step that gets you caught. Nobody is stopping you from leaving with an SD card or USB stick at Google.
Importantly, this would work way better on a speedcube than the rubick's brand cubes. Never use the rubick's brand rubicks cubes. I use the Moyu RS3m V5 maglev, and I think that it would work well for hiding uSD cards.
i'd strongly caution anybody foolish enough to go down this path
financial watchdogs and international treaties make it impossible unless you are perhaps a multi billionaire who can afford to buy people at the political level
Lying about your assets to avoid paying a lawful fine is criminal. Just because they can’t see your money doesn’t mean they can’t prove that you have it, and can’t jail you for hiding it to get out paying a fine.
Doubtful that random employees just have access to the full archive. And among those few that do have access there are probably automated systems that will catch you once you start downloading even a small percent of the content
Up to 500k for OPSEC failures is interesting, as well. It gives me hope that there are wealthy individuals contributing to sharing books, or many small donations.
Quite a few textbook authors I know are paid well to be part of the whole scheme (kickbacks, forced yearly repurchase for the 'online' component of books, etc). So I think it varies a lot.
Gemini should be trained on those books already, so in theory it could regurgitate some verbatim fragments (as NYT lawsuit against OpenAI showed some time ago).
Gemini, gpt and fable are actually very good compressions of internet content. But is lossless compression as in they kept the most important part (for them to fulfill the next token task) and found a way to mimic the rest.
> Virtually all major companies building LLMs contacted us to train on our data. Most (but not all!) US-based companies reconsidered once they realized the illegal nature of our work. By contrast, Chinese firms have enthusiastically embraced our collection, apparently untroubled by its legality.
> We have given high-speed access to about 30 companies. Most of them are LLM companies, and some are data brokers, who will resell our collection. Most are Chinese, though we’ve also worked with companies from the US, Europe, Russia, South Korea, and Japan. DeepSeek admitted that an earlier version was trained on part of our collection, though they’re tight-lipped about their latest model (probably also trained on our data though).
It's at least 30 companies, each of which paid hundreds of thousands of dollars.
The US should just find a way to quietly share literature access with the Russians, rather than letting piracy be promoted and facilitated for US consumers as freedom-fighter "archiving".
Between all the piracy, and all the AI training and the purchase/visitor-circumventing AI services, the practice of writing and publishing genuinely good work is being wiped out.
We're killing the goose that lays the eggs, for selfish gain.
This ship has sailed for academic publications, and academics define that term very liberally because we want to read everything, fiction included. The shadow libraries started off as a way for scholars in ex-Soviet countries in particular (but also India, SE Asia, etc.) to access literature that simply wasn’t available in their country. But the shadow libraries proved so successful and convenient that academics in all countries are using them now, even if they have access to official subscription services. I use AA several times a day and so do the researchers around me in my office; at conferences, if the presenter mentions an interesting publication, the whole room immediately opens AA on their laptops, etc.
Even if projects like AA didn’t have nation-level support, academics would find a way to keep as much of it as possible going. After all, we’re the ones who compiled the bulk of pre-2020 material, and we’re the ones who do all the hard work of scanning from our institutional libraries stuff that doesn’t exist anywhere in digital form.
>the practice of writing and publishing genuinely good work is being wiped out.
Most of the best literature in the English language was written before modern IP law was even a thing. There's very little good literature written by authors primarily motivated by money.
Well, you needed the means to get an education, since most of the poor in those days were illiterate, which is something of an impediment to becoming a successful writer.
I can only think of one writer off hand who wasn’t a wealthy landowner, although it is a particularly notable example; that of William Shakespeare.
Shakespeare wasn’t poor (his parents seem to be of upper middle class standing), he was able to get a basic (but not a university) education and then pursue an acting career (with perhaps a side hustle as a teacher). Whatever the case he certainly wasn’t independently wealthy before he started writing, he needed to earn a living.
He did seem to be in it for the money (and fame) since he wasn’t just a writer he was an actor, theatre owner, and something of a celebrity, and he did make enough money to become a wealthy landowner by the time he died.
That's just cultural elitism. I hope you meet someone in your life who finds absolute joy in reading young adult romance novels or D&D fantasy books so you can understand how irrelevant "good" literature is. I love Dostoevsky and Verne (and D&D novels, especially those written by R.A. Salvatore), but I would never judge the modern "IPs" that got my daughter into reading.
Everyone has their own opinion as to what the best literature is, just like what the best music is.
But there is also some consensus. For music it would be Beethoven, Mozart, the Beatles, etc.
For literature it would include Tolstoy, Shakespeare, Cervantes, Dickens, Austen, Tolkien, and many more. I would bet it will eventually include Stephen King but it's too early to make that call now.
Possibly but this act of governmental self-harm is useful to The People. We live in a world where if your valuation is ~1T you can more or less just do what you like. And the work of The People is stolen from you and launderd.
In such a world, isnt it useful that governments are stupid enough to give adversaries reasons to undermine it? When the government props up a corporate tyranny domestically, and racketeering, should we make a temporary alliance with all its enemies?
(Eg., the provision to AI companies of all corporate secretes and competitive practices via prompts, eventually to be used against their capital interests and their labour interests).
>We're killing the goose that lays the eggs, for selfish gain
We already did that when the internet collectively agreed decades ago that everything digital should be free for anyone.
We're now 20 years downstream of ad-blocking being a virtuous good, and piracy being the ultimate show of liberty, and now suddenly everyone cares about the creator's revenue stream.
The mask slipped and unsurprisingly the internet is a bunch of selfish morally stunted children. Some of them even pushing 50 years old.
Yes, I am talking to you with the 4TB of pirated content, proud of not loading any ads in the last 15 years, and getting enraged over LLM training.
> Yes, I am talking to you with the 4TB of pirated content, proud of not loading any ads in the last 15 years, and getting enraged over LLM training.
That's oddly-specific :-)
In any case, I have no pirated content that I know off, neither proud nor ashamed of blocking ads[1], but I still get annoyed that a bunch of VCs can use their invested-into companies to launder all the worlds IP, then sell it back to them.
[1] Who feels proud of blocking ads? It's like feeling proud of tying your shoelaces: "Good job, well done, but that's the expectation, son".
AA compiles from everywhere; LibGen and Z-Lib served as the major sources of books. This has unfortunately led to search results for a particular book containing multiple versions of that book, and it is not readily clear which one is the highest-quality version. A real library would have librarian staff who carefully curate everything, but in the pirate world this isn’t realistic so it just gets all thrown together.
LibGen is now more or less a dead project. The servers of the original version were reportedly seized a couple of years ago already, and other sites under the LibGen name were notorious for piggybacking the original collection and just plastering it with ads. If one wants to upload stuff, better now to upload it to Z-Lib (not a perfect site, but still) and it will then get picked up by AA in a few months.
Oh, I have no expectation that Page or Brin would do something like this, let alone do it openly. But Page did seem to care about Books access at one point, and I find the image of him secretly leaking the Google Books corpus amusing.
How is Anna's Archive funded? I see they have memberships, but it's hard to believe that can fund all these bounties - some going into six figures. Ask any FOSS project about funding by that method.
It seems like there are some deep pockets funding them.
On another note, if Google's cybersecurity were always one rogue employee away from a massive leak, then it wouldn't be Google. What was the last Google leak you remember, defense in depth people.
If it works as AA seems to theorize, you'd need to:
(a) work out how Google books exposes fragments of books, and see if there's a systematic way of using this to get whole books. For example, a naive approach might be to find any fragment of the book by searching some exact phrase. Then, you can search for an exact phrase from the start or end of the fragment it gave you, hoping it will show you the previous or next part of the book. You can then just loop that to get the whole book.
(b) once you have (a), you need a way of bypassing Google's bot detection/rate limiting. I don't know what current state of the art is, but there may be a solution for sale out there. E.g. you pay to receive a cookie or browser state, and use that to fetch the URLs from (a). Or if you're good/already in the scene, you could do this part yourself.
Another explanation might be a general dislike of big establishments like AI companies and publishers (which glosses over individual authors, but they probably make up a negligible portion of total sales anyway).
The problem is that it is quite difficult to access the published papers is you are not in academia or some company that pays for the access, so AA sort of serves that niche to transfer the knowledge. Training on the other hand is a commercial activity to later rent the model, if this would be purely for open weights I suspect everyone was cool with it.
That’s misunderstanding why these models are behind. A large part of why they’re behind is they aren’t able to do the reinforcement learning post-training steps that takes a pre-trained model and turns it into a frontier model like GPT 5 or Opus. Instead they do their best to recreate these models using distillation.
Fundamentally, you can never distill your way to being the teacher, so these approaches will not advance the frontier.
[edit, after thinking about it I think my phrasing is unfair. It's not necessarily that aren't able to do it, but they haven't yet shown that they are willing to do it.]
That’s not remotely true. They did distillation as a cheap solution to the cold start problem. You need data/trajectories to hill climb to higher capabilities. All large Chinese labs do RLAIF.
Oh yes, not remotely true. Which is why the frontier labs all have invested heavily in trying to identify and thwart distillers, using known company names / domains to drive their exclusion lists.
It's cheaper to distill than to do reinforcement learning, so of course they prefer that, but if it wasn't an option they could just pay up and spend more GPU time on RL.
>"they aren’t able to do the reinforcement learning post-training steps"
Not yet.
If there is a need someone will come and fulfill. Personally for me now I do not even want to use top models. Professionally I use AI to help with the coding using Junie agent that comes with IDEs from JetBrains. Junie is told to use Gemini Flash and works fine for what I ("I" being an emphasis here) ask it to do. I tried more advanced models and different vendors only to discover credits going down the toilet without any extra benefit.
Chinese companies giving away expensive models for free is a symptom of the AI bubble, too. It's not a law of nature that they'll always be able to scrounge up the money for yet another training run.
Shaping the tool that does the thinking is quite valuable when you're in the business of changing how people think - I think we can expect propaganda agencies to be subsidizing model creation forever.
This doesn't strike me as a symptom of a bubble - except in so far as the bubble pushes the competitors models forwards and thus they need to invest more to stay competitive.
If you think Chinese companies always act as a bloc, your mental model needs to get about a billion times more detailed. But in this case just a few details may be enough: There are Chinese AI companies that have released LLMs without publishing the weights.
ByteDance is going the direct-to-consumer route with their Doubao chatbot (the most popular in China, probably thanks to their social media prowess). iFlyTek seems to be angling for enterprise and government use cases, where they already have an in.
The companies that have released weights have in common that they didn't have a monetization channel lined up and their models weren't good enough to make people pay attention with just API access. (You can see with Qwen Max that the calculus can change towards not releasing weights for better models.)
And who exactly among the investors is having their complement commoditized? When Nvidia releases Nemotron, the story is clear, but it's less obvious for say Z.ai's GLM.
Internet was a bubble, so was telecom etc. at some point. Being bubble does not mean that when 90% of investments go down the drain the remains are not useful.
"The Internet" was not a bubble. Companies with no long-term business model / sufficient product-market fit that were riding hype were the "dotcom bubble". But when those companies crashed, nobody said "I really want to get my hands on their IP", because it wasn't valuable – an important pre-requisite to the the bubble popping. Seems to be a different case here if people actually want the SOTA models.
And why's that? I don't see how either of those two things relate to one another. A grape is a naturally occurring fruit, while the creation of a book is wholly a human creative endeavor.
There’s a lot of crops that are very intensively bred to create new, specific cultivars. Take a look at all the new Apple varieties popping up. There’s nothing natural about their creation: https://applerankings.com
When I buy grapes, yeah, I get that someone worked for me to get 'em. But I don't get all metaphysical and spooky about it. I'm buying the grapes as objects. Like a book. Once I take ownership of the object, yes, thanks for your work. I paid you and I wish you well.
As far as what I ought to be able to do with the thing? Beat it. Of course there are exceptions. You say, "Don't scan the book and upload it to object storage and place it behind a CDN so Ahmed from Tounis can access it". I say, "Don't make wine".
If it were not for Anna's Archive and Z-Library, I would've never been able to read the books that shaped who I am today, or keep my passion for learning alive.
Thanks, AA and ZLib! (Also, thank you to the authors whose books and knowledge I consumed without being able to pay them back.)
I’m kidding. Knowledge should be free. It was never created in a vacuum. It belongs to us all
This is a common perspective on HN, but it's so jarring. Someone violates an open-source license and we grab our pitchforks. Someone pirates books and it's fine - really, the authors should be thanking us.
Good books are incredibly challenging to write, more so than good software. It's not like you grab Harry Potter and say "I'm just gonna change character names and rephrase some of the text". Most authors recognize that not everyone can afford books and then contend that some amount of sharing is healthy, no different from borrowing books from a local library. If you ask nicely, they will probably send you a PDF for free. But the scale of online book piracy is absolutely staggering and demoralizing, and most of it has nothing to do with taking any serious moral stance. It's just "lol, why pay when you can download for free".
As opposed to that, books, movies are pirated for personal consumption. Not monetary gains. If someone bought a $30 book, and then ran a BaaS with millions of VC money in his pocket, people in HN would be angry at him, too.
I respect copyright, but I can’t respect a lockup period that can push to 130 years or more. For example: if JK Rowling is alive in 4 years the first Harry Potter book will have a valid copyright extending from the 20th, 21st and into the 22nd century. Is it really defensible to say that your great, great (great?) grandchildren should benefit from a government mandated monopoly on your work?
Kind of weird too think about it that way, but food for thought.
100+ year copyrights only help the descendants of the people that have already become extraordinarily wealthy. Cutting off at 2 decades would have close to 0 effect on the huge majority of creatives, while benefitting society immensely. JK Rowling would still be a billionaire, and small authors would be fine too.
If you look back through the annals of Free Software, one often encounters the claim that the GPL was a way to use copyright against itself, and if there were no more copyright, there would be little need for these licenses.
There needs to be a middle ground, such as: after 15 years of publication any private individual can access and read the work for free, but the rightsholder still controls commercial sales, merchandising, licensing, character rights, movie rights etc.
There is fair use doctrine.
And most countries have a copyright limit of 70 years after the death of the latest living author.
Also, private individual piracy of copyrighted works goes largely unprosecuted.
What more do you communist fuckwits want to take from other people?
Imagine you're a professional writer and it's your main source of income. How would you feel if someone said this to you? Would you still want to write books?
So it’s ok to take what they produced without paying for it? What a weird non sequitur.
Someday, some pro-piracy lunatic will come up with something vaguely coherent that isn’t just “me want, me get.”
"What an inestimable advantage it would be, if, in every branch of literature, there existed only a few but excellent books! This can never come to pass so long as money is to be made by writing." --Arthur Schopenhauer
Can I move in and shag your wife too?
Cheers, love your work bro.
I can't find the post but years ago on Reddit an author posted stats showing when her book turned up pirates online, real sales for it collapsed.
Because of this I make a point of buying books, programming books especially. Yes I download pdfs, I use them as previews. This has led to buying way more than I would have.
Anyway, I appreciate this doesn't apply if you live somewhere that these books can't be purchased. But everyone praising these sorts of sites tends to look at them from only a positive perspective.
I remember opening "thinking fast and slow" and noticing the weird paging. After checking the official version's page count and seeing how the version in my hand doesn't match, my best guess was that someone had printed a pirated epub version.
I'm not part of the market for these products. I don't have access to them nor even if I buy some imported (probably illegally and by single persons) or printed version, am I going to benefit them since I'm disconnected monetarily from yhe author.
Me reading pirated versions of these books has no negative effects on the earnings of the authors.
I think that's at least a bit debatable. People thought that about (normal) libraries back in the day, but it ended up having the opposite effect.
Not to mention out of print books or academic books which is a big usage of sites like these, since lots of people prefer physical books and only reach for pdfs as a last resort.
(Archiving culture alone is not the same as also enabling universal access to the culture and knowledge one is acting as custodian for and serving to global citizens)
The Internet Archive has lost its appeal in Hachette vs. Internet Archive - https://news.ycombinator.com/item?id=41447758 - September 2024 (793 comments)
https://archive.org/details/brewsterkahlelongnowfoundation
Totally unrelated: Dweb camp 2026 is coming up for those interested: https://dwebcamp.org/
(no affiliation with any person or entity mentioned in this comment)
This is key for getting epubs to your Kobo.
I've been using MoonReader for many years now and settled on pretty good parameters that make the reading experience very comfortable on both my phone and my tablet.
https://github.com/ZlibraryKO/zlibrary.koplugin
If you mean stripping drm I used Calibre for that but mostly I just avoid buying books with drm where possible.
Not sure if we will qualify for a bounty, but happy to share! Btw, we are looking for funding from small or large donors who want to help us translate the Renaissance…
I can't quickly tell what all you have archived^, but I have some friends who are academic historians who might be interested in certain categories of work (and could help verify some esoteric languages) - is it possible to search by region or language?
Have you reached out to any types of historians WRT the project? It seems like some PhD students might be able to find some projects in this work etc
^ when I looked at the timeline https://sourcelibrary.org/timeline, I got an error
Please share with historian friends. I’m not great at socials or fundraising but this was really designed to support humanists. It can give DOIs for the versions of the translated books, which means they can be quoted and cited in academic papers.
Tip: Try it in Claude or Claude code (even better)! Just point it towards the source library. It can find quotes and evidence on any topic of interest. Or try the librarian — our source-grounded research agent https://sourcelibrary.org/librarian
Thanks for the feedback, I’ll fix the timeline.
You can add it up!
I have seen Gemini costs change quite a bit when processing very similar books from the same series lately, mainly because thinking tokens have increased about 5x. Has that has happened to you as well?
Edit: for ocr I am using about 15k-25k tokens per page, but I have a complex prompt.
Cloudflare captchas have made the internet unusable for me, and I'm sure it will only get worse over time. I'd much rather just browse (or even torrent) a copy of archive.is or similar. The latter would be much better for privacy, and hey, I run ad blockers anyway.
Well, there is this little conflict of interest
https://redlib.catsarch.com/r/Annas_Archive/comments/1f6h74r...
https://reddit.com/r/Annas_Archive/comments/1f6h74r/im_curio...
True or not I rather like that one.
This one's good as well. This is an example of a rare good Reddit thread.
Apparently this is the reference: https://www.youtube.com/watch?v=FKCmyiljKo0
But this is what I thought of: https://youtu.be/hdsB99HQ2S8?t=194
All are examples of people with less power protecting each-other from a more powerful force.
And then you could obviously just buy it physically where buying is definitely owning, so I find that sentence a bit inappropriate for books
But I could be wrong.
I am more surprised to see that there are so few alternatives to it. Or perhaps I am unaware of them but after Facebook and co declared war on libgen, and libgen going down, there were surprisingly few alternatives. Anna was one of the few. I still don't know what happened with libgen, but since the attack it really is kind of semi-gone.
And even fewer who are single and childless. (Google would likely go after the estate of anyone who did this.)
This is something they’d want to settle quietly, so the family would have leverage.
The book publishers might actually be the bigger problem. They'd have civil copyright infringement claims with giant statutory damages.
financial watchdogs and international treaties make it impossible unless you are perhaps a multi billionaire who can afford to buy people at the political level
Lying about your assets to avoid paying a lawful fine is criminal. Just because they can’t see your money doesn’t mean they can’t prove that you have it, and can’t jail you for hiding it to get out paying a fine.
[this] appears as a link to a .li address, and that goes bad places.
Should be https://annas-archive.gl/volunteering#bounties
> Purchase all Library of Congress MARC datasets — $3,000 bounty
> English Wikipedia pages about relevant institutions — up to $100 per new page
> Internet Archive Digital Lending — $5000 per 1 million pdf files
> Text version of our full library — $20,000
...
https://software.annas-archive.gl/AnnaArchivist/annas-archiv...
The current situation feels untenable with renting. So many regular people I know have learned about VPN, NAS, etc.
Or something like thanks.dev
Spotify, Netflix, Amazon etc provided OK value for a while, but now enshitification is biting, this is due a massive comeback.
https://annas-archive.gl/blog/ai-copyright.html
> Virtually all major companies building LLMs contacted us to train on our data. Most (but not all!) US-based companies reconsidered once they realized the illegal nature of our work. By contrast, Chinese firms have enthusiastically embraced our collection, apparently untroubled by its legality.
> We have given high-speed access to about 30 companies. Most of them are LLM companies, and some are data brokers, who will resell our collection. Most are Chinese, though we’ve also worked with companies from the US, Europe, Russia, South Korea, and Japan. DeepSeek admitted that an earlier version was trained on part of our collection, though they’re tight-lipped about their latest model (probably also trained on our data though).
It's at least 30 companies, each of which paid hundreds of thousands of dollars.
Between all the piracy, and all the AI training and the purchase/visitor-circumventing AI services, the practice of writing and publishing genuinely good work is being wiped out.
We're killing the goose that lays the eggs, for selfish gain.
Even if projects like AA didn’t have nation-level support, academics would find a way to keep as much of it as possible going. After all, we’re the ones who compiled the bulk of pre-2020 material, and we’re the ones who do all the hard work of scanning from our institutional libraries stuff that doesn’t exist anywhere in digital form.
Most of the best literature in the English language was written before modern IP law was even a thing. There's very little good literature written by authors primarily motivated by money.
I can only think of one writer off hand who wasn’t a wealthy landowner, although it is a particularly notable example; that of William Shakespeare.
Shakespeare wasn’t poor (his parents seem to be of upper middle class standing), he was able to get a basic (but not a university) education and then pursue an acting career (with perhaps a side hustle as a teacher). Whatever the case he certainly wasn’t independently wealthy before he started writing, he needed to earn a living.
He did seem to be in it for the money (and fame) since he wasn’t just a writer he was an actor, theatre owner, and something of a celebrity, and he did make enough money to become a wealthy landowner by the time he died.
> best literature
What does that even mean?
But there is also some consensus. For music it would be Beethoven, Mozart, the Beatles, etc.
For literature it would include Tolstoy, Shakespeare, Cervantes, Dickens, Austen, Tolkien, and many more. I would bet it will eventually include Stephen King but it's too early to make that call now.
In such a world, isnt it useful that governments are stupid enough to give adversaries reasons to undermine it? When the government props up a corporate tyranny domestically, and racketeering, should we make a temporary alliance with all its enemies?
(Eg., the provision to AI companies of all corporate secretes and competitive practices via prompts, eventually to be used against their capital interests and their labour interests).
We already did that when the internet collectively agreed decades ago that everything digital should be free for anyone.
We're now 20 years downstream of ad-blocking being a virtuous good, and piracy being the ultimate show of liberty, and now suddenly everyone cares about the creator's revenue stream.
The mask slipped and unsurprisingly the internet is a bunch of selfish morally stunted children. Some of them even pushing 50 years old.
Yes, I am talking to you with the 4TB of pirated content, proud of not loading any ads in the last 15 years, and getting enraged over LLM training.
That's oddly-specific :-)
In any case, I have no pirated content that I know off, neither proud nor ashamed of blocking ads[1], but I still get annoyed that a bunch of VCs can use their invested-into companies to launder all the worlds IP, then sell it back to them.
[1] Who feels proud of blocking ads? It's like feeling proud of tying your shoelaces: "Good job, well done, but that's the expectation, son".
LibGen is now more or less a dead project. The servers of the original version were reportedly seized a couple of years ago already, and other sites under the LibGen name were notorious for piggybacking the original collection and just plastering it with ads. If one wants to upload stuff, better now to upload it to Z-Lib (not a perfect site, but still) and it will then get picked up by AA in a few months.
It seems like there are some deep pockets funding them.
https://chatgpt.com/share/6a4970e8-7fe8-83e9-8f81-3aefd76b6b...
On another note, if Google's cybersecurity were always one rogue employee away from a massive leak, then it wouldn't be Google. What was the last Google leak you remember, defense in depth people.
Training on copyrighted material
--> bad
Actually distributing copyrighted material
--> good
Needless to say, this is backwards. Any copyright holder will be much more worried about the latter.
Another explanation might be a general dislike of big establishments like AI companies and publishers (which glosses over individual authors, but they probably make up a negligible portion of total sales anyway).
Copyright reform is necessary for national security
https://annas-archive.gl/blog/ai-copyright.html
Piracy helps people who can't afford to pay or have no way of acquiring legally.
LLM training helps megacorps replace people.
If you side with common people against megacorops, you're okay with piracy and against LLM training on copyrighted works.
Fundamentally, you can never distill your way to being the teacher, so these approaches will not advance the frontier.
[edit, after thinking about it I think my phrasing is unfair. It's not necessarily that aren't able to do it, but they haven't yet shown that they are willing to do it.]
/s
Are you sure?
What if you distill from 10 teachers?
Not yet.
If there is a need someone will come and fulfill. Personally for me now I do not even want to use top models. Professionally I use AI to help with the coding using Junie agent that comes with IDEs from JetBrains. Junie is told to use Gemini Flash and works fine for what I ("I" being an emphasis here) ask it to do. I tried more advanced models and different vendors only to discover credits going down the toilet without any extra benefit.
This doesn't strike me as a symptom of a bubble - except in so far as the bubble pushes the competitors models forwards and thus they need to invest more to stay competitive.
They all carry political weights, because humans behind defend their interests, and are promoting some social values.
https://pastebin.com/hjhvsBFg
This answer from Claude is so biased that it is ridiculous
China acts like an entire bloc, not as single companies, and they want to monetize hardware.
ByteDance is going the direct-to-consumer route with their Doubao chatbot (the most popular in China, probably thanks to their social media prowess). iFlyTek seems to be angling for enterprise and government use cases, where they already have an in.
The companies that have released weights have in common that they didn't have a monetization channel lined up and their models weren't good enough to make people pay attention with just API access. (You can see with Qwen Max that the calculus can change towards not releasing weights for better models.)
And who exactly among the investors is having their complement commoditized? When Nvidia releases Nemotron, the story is clear, but it's less obvious for say Z.ai's GLM.
Lots of companies will pick them up for scrap metal prices and host them for fractions of what we are paying today.
That's the nature of bubbles.
As far as what I ought to be able to do with the thing? Beat it. Of course there are exceptions. You say, "Don't scan the book and upload it to object storage and place it behind a CDN so Ahmed from Tounis can access it". I say, "Don't make wine".