As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification
I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites.
There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.
(Cleaned / decoded: 'You are Google Search from 2004. Given a search request, provide 10 links to relevant pages, each with a short text from the page, featuring the search terms. Avoid any pages that do not contain the search terms. the search request: %s')
Steep at 300 searches per month or about 10 per day. If you search often, it is too expensive. If you rarely search, not worth 5 bucks.
They are plainly trying to push people to their $10 unlimited plan. I would have appreciated if they allowed, say, 600 searches per month for $5 or so.
Good workaround! It's just sad that to achieve the previous behavior we now need to burn significantly more compute, and in turn energy, and with far worse performance and an inverted UX.
The most frustrating part is that they have all of the data, and oodles of compute, available to surface the same very functional experience they originally offered, but they would prefer the image of being visionaries rather than the reality of being useful.
For Google, the business side of you paying them doesn't work out as obvious as it might seem.
Advertisers are sold the idea that 1000 clicks/impression is X dollars.
The average impression might be priced at 1000/X$.
But you are worth much more than that for two reasons.
- you are a person willing to pay 5$
- Google can sell a “collectivized” product. The same way that eg collective farmer crop insurance creates actual value.
and the third reason (as consequence of the second) they dont want people to think about: it gives them more opportunity to fudge the value by making it all very complicated.
I turn it down straight away for being Russian and I'm suspicious of anyone that mentions it positively.
Russia is in a cold war with democracies in general and a hot war with Ukraine. It has an autocratic dictator as leader and state control of media.
It has a well documented propaganda campaign with the goal of destabilising other countries through agitation. We even know the address of the building in Saint Petersburg it's run from.
Why would anyone try to assess Yandex's results? They could be like a slot machine serving good results 78% of the time and slipping propaganda in when least expected.
Why bother with that when there are good, free alternatives?
Couldn't we say the exact same thing about any US service in this day and age?
Trade war with half the planet, hot war with Iran. Autocratic dictator as leader and complicit media. Well documented propaganda campaign and history of destabilising countries around the world. Known to compel tech companies to spy on foreign citizens...
I have, and it has never been good for my purposes. In English, I just can't find things I need, in Russian, I find things I wish to never see again, tbh.
What is your experience with it?
Edit: The best search for my purposes historically has been on Pinterest. Unfortunately, not anymore. I normally search for visual artifacts (photos, art, schematics, etc), out of all search engines Google is still the only one that is somewhat functional for me in this regard.
I tried DDG, Kagi, Yandex. None of them worked for me.
You may not even appreciate the full extent of it. Long ago now, when Google took off and started dominating because it was “the best results” people didn’t realize that even though they were the best results, they were not actually good results. To be the best, you just have to be notably better than your nearest competitor, you don’t actually have to be good or even decent, just noticeably better. Today Google really has no competitors besides all the derivatives it has maintained to maintain an illusion of competition, the most obvious examples being DickDuckGo and Bing.
It’s somewhat similar with AI today, only the competitive landscape has majorly shifted with the Chinese models, something that wasn’t supposed to happen. All the sudden you no longer have a closed system of American models that can all be contained and managed as good enough to make people believe are “the best”; they actually have to compete in a far more competitive arena with the tension of the best model being the most competent model, and the most competent model, by virtue of AI, will be the least restricted, the least censored, the least guardrailed, the least controlled to present a telescreen and Hollywood type fantasy world where the good guys always win and … wouldn’t you know it … the good guys is always us, the people controlled by a psychopathic, narcissistic cabal that also controls what AI will tell you, the false truth.
Which being public wealth, should be publicly available.
Data that were sourced from the public, should be public. We need this legislated, or wealth inequality and fiefdoms will keep growing with the historically known outcomes.
This is interesting. Do you consider Yandex absolutely safe to use (wrt their possible connections with the Russian government, and current war in Ukraine)?
Asking this as a geeky end-user from Estonia -- I wonder if I would be considered "suspicios" by local internet service providers, our govt/police etc if I did a lot of searching via Yandex instead of e.g. Google these days.
(For us in Estonia, I suppose "everything with possible Russian ties" seems suspicious these days, unfortunately with a reason. As a side effect, this also creates a hesitation to look into interesting projects like ReactOS, the old-dos.ru repository, etc. Russia, obviously, inhabits lots of extremely talented coders who still have the skills and mindset to push older hardware to its limits, actually still write stuff in assembly for that, use the reviving DOS distros, etc -- lots of very interesting, creative coding done there, I guess. But, I hesitate visiting their webpages due to Putin's aggression in Ukraine, them being pro-Putin, and possible spying related to all this. It's a sad state of things actually. Sort of like a loss of computer-cultural ties.)
People have this fixation on privacy, when there's so much more going on, potentially way mor important.
Google's strong position in AI gives them bragging right to attract more companies and get them to actually pay, subsidizing your use. It also lowers competitors position as you're not touching them while we're on Gemini. It also fortifies their position in the future ad market.
Being second or third in AI usage is worth a lot, one's private data matters very little in comparison.
Plus soon we'll be paying based on secret parasocial influencing agendas.
It's now possible to put a complex spin/bias on whatever the system shows (or whatever it chooses to bury) in a really easy and scalable way.
Ex: Dairy Association pays, and suddenly results about bones are just a bit more likely to show something about the importance of calcium that many people get from milk. Fraternal Order of Police gets involved, and now "shot by police" everywhere gradually morphs into "was in an officer-involved shooting".
... And of course "Google making it harder to see URLs" becomes "Google taking bold steps against evil scrapers."
Me too. I don't connect my phone to my Google account. I understand that the Play store is enticing, but there are other options. Some of which are actually better.
Mmmh this change doesn't seem about gaming the index but rather about scraping the index. Which affects in a negative way only Google. And actually would affect users in a positive way because it lets other companies create their indexes more easily (at the expense of a poor mega-corp, indeed)
> Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters".
As long as you don't search anything related to Russia itself, or to Russian interests elsewhere (like their invasion of Ukraine). Then it's very heavily biased, priority is given to state ran propaganda mills, independent media is hidden from results, etc.
And before anyone starts thinking about whatabouting, Yandex are based in a country where journalists are openly and publicly assasinated to intimadate. It would be delusional to expect any sort of press and related (like search engine or aggregation) freedom, or to try to compare this to anything in any other developed country.
A Soviet citizen and an American are sitting next to each other on a commercial flight.
The American turns to the Russian and says, "I have to hand it to you — your state propaganda is very impressive. You really know how to shape what people think."
The Soviet smiles, nods, and replies, "Thank you, but it's really nothing compared to American propaganda."
The American looks shocked and says, "What are you talking about? We don't have any propaganda in America."
So, I decided to check, and the first links that show up when I search for "war ukraine" in yandex.com are Google news are from rbc.ua and bbc.com. When I search for it in yandex.com, the first link is CNN, the second google news, the third is indeed TASS, but the fourth is novayagazeta.
I then picked "bucha" as the query most likely the be affected by propaganda and censorship. In yandex.ru, there are propaganda outfits in the result (something called ruwiki.ru comes third), most links are of fairly direct accounts of the massacre by independent media. The AI summary says the town was "occupied" by the Russian army and "liberated" by the Ukrainian one and "После отступления российских войск в городе были обнаружены многочисленные свидетельства массовых убийств мирных жителей." On yandex.com on the other hand, the first page of results does look fairly propagandistic, including "globalresearch.ca" and "donbass-insider.com", which seem like propaganda outfits. But it also does include accounts from novayagazeta, al Jazeera a video of killings from Radio Free Europe.
While they do put either own propaganda, neither Western, nor Ukraininan nor independent media is not hidden from the results and it's not visibly de-prioritized.
Of course, I wouldn't discount the possibility that the results would look quite different from Russian territory. And, if anything, half the reason why Russian propaganda is so effective is that its self-aware and capable of subtlety when needed. "Fuck you if you can't handle the truth, this version of Biden is the best version ever." would never happen there. Unlike Scarborough, Soloviev knows exactly what he is.
Open and constant suppression of information is not the regime's usual strategy in information space (though, they'd ratchet up the level of control when they deem it necessary), which is why I knew the claims here wouldn't stand up to scrutiny.
I explicitly say, that I won't be surprised if the results change from Russian territory. I already put way too much effort to check the validity of an internet comment, but if anyone is interested, they can check.
>It would be delusional to expect any sort of press and related (like search engine or aggregation) freedom, or to try to compare this to anything in any other developed country.
Israel is responsible for killing more journalists than any other country. Google has profited by providing services to Israel in the ongoing genocide.
I was waiting to see how long it took for someone to "make this about Palestine" - their strategy is to inundate all discussion with the topic.
Over half the journalists in the Gaza Strip are affiliated with Hamas. Many of them infiltrated Israel in 2023, and some of them held Israeli hostages in their homes, including children hostages.
there was a source code leak and you can see some examples there which make the president's image more alluring. e.g. if you search for "Putin" they add an invisible "-crab" to your query
that's the most innocent example from the top of my head. There probably are many more serious censorship issues
Yep, plenty, and tbf it's entirely logical. The people working at Yandex do not want find themselves dying of a nerve agent or polonium, which is a real and acute danger for people who displease the ruling regime.
> And before anyone starts thinking about whatabouting, Yandex are based in a country where journalists are openly and publicly assasinated to intimadate
Thankfully Gary Webb is here with us today to laugh at this
> There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google
This is such a strange position to take. Apple doesnt allow all sorts of apps on the appstore, but that is never said as "Apple is erasing illegal streaming". Google is simply not showing the results. They are not taking down, or banning, or doing anything to the websites.
The pain being made is: If you use a search engine as your eyes to see what exists “on the internet”, then, absolutely, whatever Google hides from its results or fails to index is “erased” from “your” experience of the internet.
I see where you are coming from but the argument would be stronger is Google Chrome refused to open illegal streaming sites etc. AFAIK, there's no such restriction.
(I dont agree to this but ...) using your argument of " If you use a X as your eyes to see what exists “on the internet” .. " - we should all be mad at Apple. I use iPhone as the primary device to access apps, and they not just hides but actively ban and cut whole swathes of developers.
If I ask Siri to give me a link to illegal streaming site and if it refuses, is that cause for concern? I'd say no. Infact, I dont expect it to give me that and I get it. Same for Google Search in my humble opinion.
I see piracy sites just fine on Google. Almost always the top result. That's why I go there to find the next domain after a previously working one gets shut down.
If you use an iphone then yes, apple is indeed "erasing" illegal streaming in the same sense that google is. We can be nostalgic for the old internet while also recognizing that at least google is well within their rights not to serve up results that fall outside the bounds of the law. (Apple not so much. When you gatekeep the hardware platform I think you're ethically obligated to act as a common carrier. Unfortunately the law doesn't require that.)
> Apple doesnt allow all sorts of apps on the appstore, but that is never said as "Apple is erasing illegal streaming"
Because they are not "erasing" anything, they are refusing to provide a platform that enables the direct (app whose intended purpose is) delivery of illegal content to you.
The difference being one will not facilitate the activity, while the other, is actively suppressing it. And thats what is meant by erased.
Your question in other comment If I ask Siri to give me a link to illegal streaming site and if it refuses, then that would be the better comparison. Then we could say "other seedy corners of the internet that have been neatly erased by [Siri]"
If the companies weren't trying to destroy trust and take everything away from everyone these things would never have much traction but now they deserve to have more than ever.
I was suspicious when they started obfuscating URLs in their own browser, then on their SERPs, and now this...
For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article.
Almost exactly a year ago, Google stopped working without JS. I stopped using Google.
Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out.
Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.
While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.
While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.
It's still kind of an open problem. There are partial solutions, but not yet really an integrated one. There is Hister: https://hister.org/ that builds some sort of drive-by index of what you're browsing anyway. And there are "true p2p" solutions like YaCy: https://yacy.net/ but it takes ages to crawl the open web (and lots of storage). There's still a lot of optimization to do in this space.
Impossible. The majority of websites firewall automated crawler traffic (because of the rise of the bots), only making exceptions for the largest search engines. There is no possibility of starting a new crawler.
It would be interesting to see a decentralised, residential collective that builds and publishes an index. There are surely enough interested people on HN alone that would be willing to run software at home to scrape a small slice of the internet.
The majority of websites try to do that but they do not catch as much traffic as they think they do. A starting point for a scraper is to run it on your home connection in an undetectable web driver framework such as zendriver.
SearXNG configured as in the OpenWebUI docs is pretty cool. My "Hello World" with a new agent framework is teaching it to use SearXNG. Hook this in as a tool and the agent can answer a lot of questions.
SearXNG is more of a metasearch, the dude who wrote it pops in on here and is working on a cool sounding project that is more like a local personal search engine, I forget the name, but I've been meaning to check it out.
Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.
> Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.
Great. On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, another few second server-side stall is introduced.
I swear that every day modern computing scenarios get slower and slower instead of snappier and snappier.
Wow - this exact same bug has been happening to me too. I gave up on troubleshooting it after the first few attempts came up with nothing, assumed it was just unique to me.
I searched for this a few months ago and found some mentions of this bug, but yes it affects me too. Can stall up to 20 seconds+ sometimes. Chrome is fine.
The link you followed when you clicked hasn't a direct link for years, decade afaik (they mangle so they can see what's followed). The page used to show the direct on the search text but now it shows some stand in for it - sometimes. You can see the direct link on the bottom of the screen when you hover - sometimes (and sometimes you see a mangled link). Sometimes the google link contains the original link in the center also[1].
The situation seems to vary from result to result even on the same page of the same search - at least on the test search I just did. You can figure out what happening to an extent but this very inconsistency seems to speak to a dystopian quality to today's information gatekeepers.
I've just checked this again using a google account where search result pages are still following the old behavior: It looks like the 'href' attribute is the direct link, and the 'ping' attribute is the /url redirect link you are referring to. So it looks like it is actually sending me to the direct link, it just also requests /url at the same time in order to log the click. This means the user was not waiting for the logging/redirect request to come back.
Which is exactly the game theory that was predicted when some browsers started ignoring <a ping> to "protect privacy". If your browser supports ping you get ping, otherwise the website gets the data anyway but with a worse user experience.
I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course.
I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms.
I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought.
I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today.
I'm really glad to have fresh alternatives (like Kagi), and can't wait for the cultural zeitgeist among developers to swing back around to valuing users as human beings and living up to the trust they place in us.
I switched to DuckDuckGo around 2018 or so, which was when I realized that Google's results quality has deteriorated so much that I won't lose anything of value by doing so. This thread is how I learn about the atrocities that Google Search is committing these days.
Which is an elaborate way to say: you don't have to live like this.
I don't. At this point I just ask the oracle in natural language. There's so many to choose from. Why spend hours wading through only vaguely related things? You can ask it for a direct answer or you can ask it to teach you about the topic.
> This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today.
I think it is both monopolization and cheap money. Something in economy favors big companies and disfavors competition.
Cheap money means that a selected company can run at loss, kill competition and then enshittify to start earning. By that time it is too late and too easy to buy or destroy smaller competing company. And you get access to funding by being charming to VC, by being the kind of sociopath they like to see.
Capitalism works when there is a competition. Not when there is an oligarchy.
It's rather frustrating that after they crawled my website to use for their AI without my consent, and using that AI to slowly strangle my traffic, they're not happy that _other people_ are doing the same to them.
>”This is what capitalism's all about. Take as much as possible and give as little as possible if you want to win. Ideally, give negative amounts.”
Your description is vague enough to encompass every system that I’m aware of, from feudalism, through to mercantilism, communism, and capitalism. It may best fit communism and feudalism, both of which are most notable for fostering negative-value-added firms.
Personally, I put a lot of effort into writing my blog posts, some of which have been really well viewed by humans. While mine is effectively a "hobbyist" site, it's nice when readers look at other blog posts as a result of reading the original one. Google's approach ensures that readers get a very narrow view of my content, so everybody expect Google loses out.
That's just Google not caring about your ISP because you live in the global south. Everyone gets this. US citizens get the exact same problem but Google whitelists their networks.
It breaks the social contract of the web. My user agent should be able to tell me where a hyperlink goes without first clicking on it, and perhaps act differently based on that information.
Now, all Google search results appear to go back to Google.
TBF this is more a browser problem than a website problem to my mind. Redirects shouldn't be handled in such a cavalier manner. I should be able to configure the browser to stop and confirm the destination any time the domain changes without direct user interaction.
Aside from the privacy concerns and inconveniences this brings to Google users, it matters for users (of services) that rely on SerpApi too. Kagi for example has started showing these goto URLs in search results.
Yeah. I might be wrong but I think they only served direct links for a relatively short time in their history, early on in the 90's and in recent years with ping, which they used to track clicks anyway. At least half of their history they used either the 302 redirects or onmousedown link rewriting (which was terrible). And I'm not even starting on AMP.
In this case they assume correctly that it is enshitification. A search engine, that doesn't understand what hypertext and the web are supposed to be is shit. There are no good reasons to not have proper links. Whoever creates such websites is having other motives or doesn't know good web development. In the case of Google employees I have to assume the former. They are not acting in the best interest of the users, and therefore enshitify Google search.
Well, I hope ultimately this will lead to even more user loss for them. Fortunately, I myself don't have to suffer due to it, because I degooglified my life.
I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look.
DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.
That somehow hurts me to hear more than anything. I remember days spent in my youth trawling through Google search for anime fandoms and homework answers. Joke used to be that beyond page 1 is the definition of desperation and in the 00s that was definitely true but I discovered a lot of cool distractions in that desperation. This feels like they killed Google Reader again.
Don't know how modern this is, but I remember being disappointed that despite Google claiming it had billions of results, you could only see a few pages' worth.
Looking now, I've just noticed they no longer give a count of results.
I don't know how global consistency is actually useful, but I can easily think of ways it is less convenient.
For example, DST starts and ends on different dates in the UK and the US. Which dates should google return when someone from either country searches for just "DST end date?" Someone lives in Orange County and searches for "Orange County Sheriffs," which one of the eight Orange Counties should google return?
These are both examples where a localized — not even customized, just localized based on IP addresses — search results will easily help reduce headaches.
I wouldn't mind that one. That a man in the US and a woman in Japan would get different results for a search for "sushi restaurant" is perfectly reasonable (even if the woman in Japan was searching in English).
It's when two people in the same neighborhood got different results for the same search that I said "wait a minute, they're personalizing search results now for ad-targeting purposes" and ditched them. The potential for them to deliberately hide things from you was too great.
When I search Python I should get the programming language but when my neighbour searches Python he should get snakes. That is a valid use of personalised search.
Good example, which is why both you and the other commenter gave it. And if that's as far as it went it wouldn't have bothered me so much; it was when conservative people I know were getting different results from liberal people for the same search terms that I started to realize the potential for an echo chamber, and decided it was a bad idea.
It's good for your mental health to be exposed to arguments you disagree with. Many times they will be bad arguments and you can dismiss them, but it's good to know what the people who disagree with you actually believe. And sometimes, just sometimes, you might see something that makes you realize that it's your own worldview that was mistaken, and adjust your worldview to better fit reality. If your search results only show you echo-chamber results that already agree with yours, you'll rarely learn the areas where you were mistaken. (And nearly everyone is mistaken about some things; the person who is right about everything is a rare creature indeed, and you should never gamble that you're that person).
The age of internet search is over. The age of Cloudflare has begun. It wouldn't be possible to build a search engine now if they wanted to... and there wouldn't be anything to search for anyway. The non-corporate internet withered into dust and blew away in the wind.
If you could find what you want, how would they ever sell you what they want you to buy? And I'm not just talking merchandise, though that too. Your political narratives, your values, opinions, everything. And everyone likes it so much they just sit there scrolling and swiping and tapping.
I think curated directories from prehistory might make a comeback eventually, maybe combined with a search engine that indexes only the whitelisted websites. ironically, we can use LLMs to filter out AI slop by measuring the signal-to-noise ratio, which is atrocious for AI slop and ESL slop that preceded it.
I'm hoping for a future where we create static HTML pages again styled with a bit of handmade CSS, because we're so tired of bot attacks and long loading times. Then suddenly a Cloudflare network becomes absolet.
people have been putting their fuckass blogs with 0.5 visitors a day behind Cloudflare long before the increase in bot traffic. and that increase makes fuck all any difference for them anyway.
It means they're capable of burying news stories that would contradict your worldview and pushing news stories that support your pre-existing biases, leading to more engagement from you (a win from their point of view) but also burying you in an echo chamber. And unless you were in the habit of doing the occasional search in Incognito Mode, you wouldn't know. (And even then, they probably would be able to put together enough clues to figure out your identity even without your Google login cookie).
Yes, the fact that they could do it does not prove that they were doing it, not right away. I ditched them as soon as I found out that they could do it, because I was absolutely certain that eventually, they would end up doing it. And I wanted neutral search results, not biased ones, even ones biased towards my own point of view.
I subscribe to The New York Times, if I am searching for a news event, I’d appreciate a site that’s not paywalled and I trust be the first result if it is reasonable.
How u reliable personalised search results are depends on who controls the personalization and for what purpose.
I would love a search that lets me ban results from certain sites. I don't like search that's showing skewed results for reasons I don't know or control.
Becuse they are a monopolist and we can require certain things from monopolists.
Similarly, Bell Labs was kind of required to release transistor for anyone to license - it was a part of social contract that they were allowed to maintain their monopoly in exchange for releasing certain parts of technology.
Alternatively, they could be split up and their indexing division made an independent company selling to anyone on a free market.
When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off.
I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.
What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?
I’m guessing that the knowledge of the techniques to support large-scale web search have diffused out of Google - it has been a few decades after all. Not to mention that the distributed system knowledge that used to live only in Google was either published by Google or cloned in other projects, usually made by ex-Googlers. And you can now rent capacity at scales that 20 years ago required Google to build lots of their own data centers.
Also, there is now a use case for paid search APIs - LLMs and agents - that essentially didn’t exist a few years ago. Not sure why Google hasn’t leaned more into this, but probably a combination of Gemini and Ads interests have combined to view their search index as an increasingly valuable asset, when if anything it might be corrupted already by these interests and therefore be less valuable.
It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.
Not better, just much _much_ more convenient. And it doesn't have to be a frontier lab. I'm happy to ask any oracle that returns sufficiently good results which is an ever increasing number of them.
Dont worry in a year or two, Google wont even redirect you to the actual true url, instead everything will be a page with all links rewritten so all http is tunnelled thru them.
I really like the idea of turning the search index into a public utility. It is one of those natural monopoly coordination problem things. Just quasi nationalise it for economic efficiency. Ofc google can still sell adds against their own ui (like everyone else). Hopefully this move will move that idea closer to reality.
It's primarily relevant because it makes scraping search results much more expensive, solidifying Google's effective monopoly on Internet search.
Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.
This is the best explanation. They’ve been doing the same in Google News. Each entry comes not with a URL to the source, but with a hash. To resolve it, you must send requests to Google’s servers. Anyone who wants to create a list of URLs of sources automatically can therefore be blocked by Google now on two levels rather than one - the search for a list of results, and identifying the source URL for each result.
In effect, they’re removing attribution from the content they quote from other people’s websites. It would be interesting to see if courts object to that. It is one thing to crawl other people’s websites and display snippets of their work as your search results when each result is properly and transparently attributed. But if the text is quoted and the source is not there alongside it in plaintext, replaced only by a vague promise that, if you ask, we may or may not tell you where this piece of content is from, that is a very different deal.
You can opt out from Google scraping you though? In theory you can opt out of anyone scraping you (if people were well behaved). Google should get to opt out of being scraped too.
Well just as a regular user, I think it is pretty annoying because if I look up anything on Google while in Incognito Mode and hover over a search result, I can see that maybe the top result is maybe Wikipedia, or Instagram, or some other less-known website depending on what I'm searching for. Now, that's all very obfuscated because I don't actually know where I'm going to land for sure.
They were already tracking everything you click but for example if you want to send a link to someone you can't copy the link from the Google result and send it to them, you'd either send them the Google tracking link or go to the website yourself.
it sounds like it primarily matters if you are a customer of this company, one that is building a search index off of urls scrapably hardcoded (or at least so as to be easily unencodable in non-realtime, it sounds like?) inside google search result redirects. in theory, there could be noticeable consumer user impact, but ... it would have to be a pretty large theory
The security implications of this are very severe when you consider the amount of people who google government websites, banking, crypto and others. And google will happily serve you a phishing website either in ads or results.
It's to make it more difficult for scrapers. Google have been able to do all the tracking they've wanted for decades.
Think about it: if you want to set up something like jina.ai you need to build your own index or piggyback on Google. My guess is they use residential proxies to fire requests to Google then scrape the URLs.
Opaque URLs now give Google another gateway to detect circumvention of their anti-bot controls, helping them monetise their search index rather than allowing providers like jina.ai to succeed.
> the reasons are so they can track who you are and sell your profile advertising.
What? Like they weren’t doing this before? Obviously Google’s telemetry is tracking every link you click regardless; there’s no extra tracking benefit to this.
The reason they’re doing this seems to be to stop competitors from scraping their search results.
I use kagi exclusively because it works. I get the results I need every time, and it's not annoying. I haven't used Google in years. Simply no need. I really hope people keep paying kagi so they stick around.
At least they seem still provide results for my searxng instance. I mean sure, they are horrible but duckduckgo just blocks most queries (and I'm the only person using the ip / seraxng instance)...
Next i'll do is to implement tavilly, exa, tinyfish etc. as search engines for searxng. No agents, no mcp, just their search api endpoint.
> Combined with earlier moves like removing &num=100
Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.
Kagi is what I settled on a couple of years ago. I do think sometimes the fawning is over the top, but it is a solid search engine and tended to give me a bit better results than Google out of the box. The real win, though, is that you can give various sites a weight, so the search results will prefer or avoid sites according to your desires. Once I had that going, my search results tended to be much better than Google.
Yep, this. I was delighted they implemented this simple yet powerful idea. I don't even remember when I have needed to go to a second page of search results in Kagi and spammy clone sites which copy stack exchange verbatim are banned into oblivion where they belong. Fuck up once doing that shit, and you are forever out of my search results.
And then there is the very handy assistant, that I can jump into using follow-up questions to the quick AI queries one can choose to use or not to use by appending a "?". Again a simple idea, which empowers the user.
My ongoing concern with Kagi is they always seem to be focused on sidequests like their Orion browser and their LLM-powered Translate tool. Maybe that's interesting for some people, but I can't help but feel like I just want a damn search engine than works.
Yeah, that's the thing for me: filtering out the SEO crap that Google happily serves up.
Google's actual search results are a waste of time visiting, both for the mindless CEO content and the ad-laden, analytics-happy, javascript-heavy websites.
So I tend to use the AI overview. But plugging myself into the all-seeing corporate oracle, that grew on all the web's content, and now seeks to supplant it seems unseemly.
It's just sad that kagi will likely only be a fringe thing, and google will continue to promote these foul, foul, mindless websites and then supplant them with its AI.
I don’t fawn, I just pay and use-and-forget. It’s almost to the point where it’s a little mental bump when I have to make a browser use it again, setting up something new.
Kagi has a pile of features I’m not getting the benefit of too, I’m sure, because it’s just-search, mostly, to me.
That’s fine; I know what I’m supporting, and I know that I’m the customer and not the product.
Because I wouldn't want Google to know what links I click on. When I used to use Google, and they added such redirects that included the destination URL as a parameter, I'd edit the link to make it just that URL before resolving it. This new scheme would make that impossible.
I use a Firefox extension to rewrite those Google redirects into plain links, to remove Google tracking of which links I click. The extension is broken now
Blocking URL shorteners and google-ad links yes! Personally for me it's also the fact that this is effectively an unresolvable URL shortener, store that link somewhere and it will most likely be dead. Can't copy link anymore and paste it on a notepad or chat app to check it out later as there isn't a guarantee it will load at all (ie: the problem with url shorteners).
This is such an incredibly annoying and deliberate defect. I want an extension that will let me resolve the true url without visiting the site, or even rewrites the entire page to show the url.
I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?
What I see now, and it's been like this for a while, is this:
You get the results and they do have direct URLs. But then, if you do some things with the link, e.g. right click to open it in a new tab, it swaps the URL to the indirect one. The idea is that initially you see a normal link, with a normal URL which will be displayed correctly when you hover the mouse over it, but right before you click it, it's swapped for the indirect one.
So, they have been doing stuff like this for a while and it has been somewhat fluid, because the swapping can occur on different events and I have also seen it load with all the links pre-swapped to the indirect ones, sometimes.
So, yes, what you see may be different and you may get the indirect URLs swapped at different stages.
Whenever I've tried this, it seems okay if you need an answer to a question, but plain bad if I'm looking for a specific page.
e.g. I'm just now looking for the menu for a local restaurant. "restaurantname menu" in Kagi (Google would presumably be similar) returns a link to the menu as the first result in about a second. Or "restaurantname menu !" goes directly to the menu in about a second.
Meanwhile, searching "restaurantname menu" in chatgpt takes about 5 seconds to return an embedded map from mapbox showing the location of the restaurant. If I click the restaurant pin on the map, there's no menu link, the 667 reviews have no link or way to view, and the restaurant description literally says "I don't have enough information to identify which local business <restaurantname> refers to."
Below the map there's some text: "If you mean <restaurantname> in <place>, here’s the current menu. <restaurantname>". The <restaurantname> link just opens the same card as clicking the pin on the map.
After that there's a bullet point list of the menu that ommits a ton of detail and options.
After that there's finally a link... that I can click to open up a popup at the bottom of the page with an actual link to the menu.
This was literally the first thing that popped into my head, I didn't have to put any effort into finding a query where chatgpt falls on its face.
Been using Brave Search now for a while including its Brave AI and aside of sporadic times I never needed Google (albeit Brave Search is slower to Google, you get used to it)
Yeah seems pretty obvious to me that most people are not going to be using a search engine in 5 years. In the sense of searching for something and combing through the results to find the answer.
ISPs can presumably correlate the Google query string with the request following the response to the goto and so make a search index? I guess they would charge too much.
Do any large ISPs use visit data to feed into a search index?
Unless you're using DNS over HTTPS they can see the unencrypted DNS traffic.
There's also Encrypted Client Hello, but they can also see which IP you're connecting to.
Note: Article published by Autom.dev which, from a quick read of their homepage, seems like it scrapes Google search results in violation of Google's terms of service and sells those results to customers via an API. That's just my quick read of it, though, so this could be wrong.
Anecdata: I've observed this behavior for several weeks already as a regular user without a Google account and there are countless comments from regular users reporting the same behavior.
why is this such a bad thing? it's not really any different from using a uuid as a user facing key, which basically everyone does.
and trying to protect your moat isn't automatically a bad thing. they clearly feel it's helping competition, so they're closing a hole. competition is good doesn't mean help your competitors.
- coming from someone who's been using fastmail as my personal for ~10 years because i don't want my emails to be backprop fodder
I guess someone made a website which google crawled and adding a senf made uuid to it is like google trying to own it rather than just being a true search engine just having index to it.
It hasn’t provided direct URLs for decades? Not exactly new behaviour.
I’ve got something that will blow your mind. Google now has tracking analytics, for get this, your business’s phone number. Some “Adsense partner” convinced our web admin to install a little script which changes your phone number on your website so they track phone call enquires back to search engine leads / advertising spend.
Yeah no thanks, that was creepy as hell and had it rolled back ASAP. You’ve got to realise the power these tech companies hold over your business. Don’t show up in the search results, someone lists your business as closed in maps, a tracking phone number goes dead so they can’t call you, you might as well have shut up shop and ceased to exist.
Would that company then sell that data back to Google? The consolidation of control of information under a single actor is certainly a factor, but it's not that simple, and it's not the only factor. You shouldn't be so eager to outsource as much of your business intelligence as possible.
Previously the clear target URL of a search result was added to the result-link (or div?) as an additional data-tag (aria or whatever, can't remember), and thus was able to be used with a userscript or similar to replace the Googleified tracking redirect link. And it was also part of the googliefied tracking URL, either way you were able to see the clear target URL.
That no longer is possible, you must ask google.com for consent to visit the result link before seeing what the final target URL is.
But the extension doesn't need to use that exact mechanism, and the person that linked it didn't mention specific mechanisms. The purpose of the extension is putting the real URLs back, and it can still do that.
And it can still prevent google from knowing which search results you click on, even though OP didn't mention that feature.
I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites.
There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.
But LLMs can be commanded. This is may bookmark alias for invoking the spirit of old Gog=ogle from within the new, AI-based Google:
https://google.com/search?q=You%20are%20Google%20Search%20fr...
(Cleaned / decoded: 'You are Google Search from 2004. Given a search request, provide 10 links to relevant pages, each with a short text from the page, featuring the search terms. Avoid any pages that do not contain the search terms. the search request: %s')
Steep at 300 searches per month or about 10 per day. If you search often, it is too expensive. If you rarely search, not worth 5 bucks.
They are plainly trying to push people to their $10 unlimited plan. I would have appreciated if they allowed, say, 600 searches per month for $5 or so.
That is interesting, which LLMs? Asking as someone paying monthly Claude subscription.
You can even choose a model for each turn.
The most frustrating part is that they have all of the data, and oodles of compute, available to surface the same very functional experience they originally offered, but they would prefer the image of being visionaries rather than the reality of being useful.
https://www.google.com/search?q=%s&udm=web
I still support the use of Kagi though, as a market signal to Google that we'd even pay them for their product if it wasn't dogshit.
Advertisers are sold the idea that 1000 clicks/impression is X dollars. The average impression might be priced at 1000/X$.
But you are worth much more than that for two reasons.
- you are a person willing to pay 5$
- Google can sell a “collectivized” product. The same way that eg collective farmer crop insurance creates actual value.
and the third reason (as consequence of the second) they dont want people to think about: it gives them more opportunity to fudge the value by making it all very complicated.
These periodic astroturfing campaigns for a Russian product on HN are annoying, and appaling if you have any morals.
Russia is in a cold war with democracies in general and a hot war with Ukraine. It has an autocratic dictator as leader and state control of media.
It has a well documented propaganda campaign with the goal of destabilising other countries through agitation. We even know the address of the building in Saint Petersburg it's run from.
Why would anyone try to assess Yandex's results? They could be like a slot machine serving good results 78% of the time and slipping propaganda in when least expected.
Why bother with that when there are good, free alternatives?
Trade war with half the planet, hot war with Iran. Autocratic dictator as leader and complicit media. Well documented propaganda campaign and history of destabilising countries around the world. Known to compel tech companies to spy on foreign citizens...
What is your experience with it?
Edit: The best search for my purposes historically has been on Pinterest. Unfortunately, not anymore. I normally search for visual artifacts (photos, art, schematics, etc), out of all search engines Google is still the only one that is somewhat functional for me in this regard.
I tried DDG, Kagi, Yandex. None of them worked for me.
How is Yandex more morally-bankrupt than Google in your analysis?
There are "politics and ideology" and there is right and wrong.
There's no doubt about that but it definitely meets some people's needs.
IMO only useful for piracy.
I prefer Google itself but this goto update is terrible.
Don't forget cached pages.
It’s somewhat similar with AI today, only the competitive landscape has majorly shifted with the Chinese models, something that wasn’t supposed to happen. All the sudden you no longer have a closed system of American models that can all be contained and managed as good enough to make people believe are “the best”; they actually have to compete in a far more competitive arena with the tension of the best model being the most competent model, and the most competent model, by virtue of AI, will be the least restricted, the least censored, the least guardrailed, the least controlled to present a telescreen and Hollywood type fantasy world where the good guys always win and … wouldn’t you know it … the good guys is always us, the people controlled by a psychopathic, narcissistic cabal that also controls what AI will tell you, the false truth.
Data that were sourced from the public, should be public. We need this legislated, or wealth inequality and fiefdoms will keep growing with the historically known outcomes.
Asking this as a geeky end-user from Estonia -- I wonder if I would be considered "suspicios" by local internet service providers, our govt/police etc if I did a lot of searching via Yandex instead of e.g. Google these days.
(For us in Estonia, I suppose "everything with possible Russian ties" seems suspicious these days, unfortunately with a reason. As a side effect, this also creates a hesitation to look into interesting projects like ReactOS, the old-dos.ru repository, etc. Russia, obviously, inhabits lots of extremely talented coders who still have the skills and mindset to push older hardware to its limits, actually still write stuff in assembly for that, use the reviving DOS distros, etc -- lots of very interesting, creative coding done there, I guess. But, I hesitate visiting their webpages due to Putin's aggression in Ukraine, them being pro-Putin, and possible spying related to all this. It's a sad state of things actually. Sort of like a loss of computer-cultural ties.)
Just like Russia doesn’t care about online criminals being located within their borders - as long as they target outside.
While this may be true to some extent of all the countries, Russia is the one to be clearly against Estonia, Baltics and eastern europeans.
It's really not free. You're paying with your data.
Google's strong position in AI gives them bragging right to attract more companies and get them to actually pay, subsidizing your use. It also lowers competitors position as you're not touching them while we're on Gemini. It also fortifies their position in the future ad market.
Being second or third in AI usage is worth a lot, one's private data matters very little in comparison.
what does this even mean
It's now possible to put a complex spin/bias on whatever the system shows (or whatever it chooses to bury) in a really easy and scalable way.
Ex: Dairy Association pays, and suddenly results about bones are just a bit more likely to show something about the importance of calcium that many people get from milk. Fraternal Order of Police gets involved, and now "shot by police" everywhere gradually morphs into "was in an officer-involved shooting".
... And of course "Google making it harder to see URLs" becomes "Google taking bold steps against evil scrapers."
https://github.com/rumca-js/Internet-Places-Database
I hate the very idea that destination location is opaque, and user can't verify if you are funneled toward malvertizing.
I hate even base64 encoded links in the results.
Google is making it harder for people to game the index, working in your favor.
As long as you don't search anything related to Russia itself, or to Russian interests elsewhere (like their invasion of Ukraine). Then it's very heavily biased, priority is given to state ran propaganda mills, independent media is hidden from results, etc.
And before anyone starts thinking about whatabouting, Yandex are based in a country where journalists are openly and publicly assasinated to intimadate. It would be delusional to expect any sort of press and related (like search engine or aggregation) freedom, or to try to compare this to anything in any other developed country.
A Soviet citizen and an American are sitting next to each other on a commercial flight.
The American turns to the Russian and says, "I have to hand it to you — your state propaganda is very impressive. You really know how to shape what people think."
The Soviet smiles, nods, and replies, "Thank you, but it's really nothing compared to American propaganda."
The American looks shocked and says, "What are you talking about? We don't have any propaganda in America."
The Soviet smiles again and says, "Exactly"
I then picked "bucha" as the query most likely the be affected by propaganda and censorship. In yandex.ru, there are propaganda outfits in the result (something called ruwiki.ru comes third), most links are of fairly direct accounts of the massacre by independent media. The AI summary says the town was "occupied" by the Russian army and "liberated" by the Ukrainian one and "После отступления российских войск в городе были обнаружены многочисленные свидетельства массовых убийств мирных жителей." On yandex.com on the other hand, the first page of results does look fairly propagandistic, including "globalresearch.ca" and "donbass-insider.com", which seem like propaganda outfits. But it also does include accounts from novayagazeta, al Jazeera a video of killings from Radio Free Europe.
While they do put either own propaganda, neither Western, nor Ukraininan nor independent media is not hidden from the results and it's not visibly de-prioritized.
Of course, I wouldn't discount the possibility that the results would look quite different from Russian territory. And, if anything, half the reason why Russian propaganda is so effective is that its self-aware and capable of subtlety when needed. "Fuck you if you can't handle the truth, this version of Biden is the best version ever." would never happen there. Unlike Scarborough, Soloviev knows exactly what he is.
Open and constant suppression of information is not the regime's usual strategy in information space (though, they'd ratchet up the level of control when they deem it necessary), which is why I knew the claims here wouldn't stand up to scrutiny.
>It would be delusional to expect any sort of press and related (like search engine or aggregation) freedom, or to try to compare this to anything in any other developed country.
Israel is responsible for killing more journalists than any other country. Google has profited by providing services to Israel in the ongoing genocide.
https://cpj.org/special-reports/record-129-press-members-kil...
https://afsc.org/gaza-genocide-companies#Google:~:text=Googl...
Over half the journalists in the Gaza Strip are affiliated with Hamas. Many of them infiltrated Israel in 2023, and some of them held Israeli hostages in their homes, including children hostages.
https://www.terrorism-info.org.il/en/abou
that's the most innocent example from the top of my head. There probably are many more serious censorship issues
https://hal.science/hal-03217497
https://euvsdisinfo.eu/yandex-from-tech-innovation-to-inform...
https://pmc.ncbi.nlm.nih.gov/articles/PMC10130930/
https://misinforeview.hks.harvard.edu/article/a-story-of-non...
Thankfully Gary Webb is here with us today to laugh at this
This is such a strange position to take. Apple doesnt allow all sorts of apps on the appstore, but that is never said as "Apple is erasing illegal streaming". Google is simply not showing the results. They are not taking down, or banning, or doing anything to the websites.
(I dont agree to this but ...) using your argument of " If you use a X as your eyes to see what exists “on the internet” .. " - we should all be mad at Apple. I use iPhone as the primary device to access apps, and they not just hides but actively ban and cut whole swathes of developers.
If I ask Siri to give me a link to illegal streaming site and if it refuses, is that cause for concern? I'd say no. Infact, I dont expect it to give me that and I get it. Same for Google Search in my humble opinion.
Why would it be stronger? Google has a near-monopoly on search.
> If I ask Siri to give me a link to illegal streaming site and if it refuses, is that cause for concern?
Yes, because you cannot replace Siri on a device you own.
Because they are not "erasing" anything, they are refusing to provide a platform that enables the direct (app whose intended purpose is) delivery of illegal content to you.
The difference being one will not facilitate the activity, while the other, is actively suppressing it. And thats what is meant by erased.
Your question in other comment If I ask Siri to give me a link to illegal streaming site and if it refuses, then that would be the better comparison. Then we could say "other seedy corners of the internet that have been neatly erased by [Siri]"
For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article.
Almost exactly a year ago, Google stopped working without JS. I stopped using Google.
Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out.
Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.
While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.
SearXNG is more of a metasearch, the dude who wrote it pops in on here and is working on a cool sounding project that is more like a local personal search engine, I forget the name, but I've been meaning to check it out.
There is also Common Crawl.
https://docs.openwebui.com/features/chat-conversations/web-s...
https://github.com/brian-learns/xng-agent
It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.
The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL.
Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.
Great. On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, another few second server-side stall is introduced.
I swear that every day modern computing scenarios get slower and slower instead of snappier and snappier.
Have you tried disabling HTTP/2?
The situation seems to vary from result to result even on the same page of the same search - at least on the test search I just did. You can figure out what happening to an extent but this very inconsistency seems to speak to a dystopian quality to today's information gatekeepers.
[1] Example. https://www.google.com/url?sa=t&source=web&rct=j&opi=8997844...
I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms.
I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought.
I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today.
I'm really glad to have fresh alternatives (like Kagi), and can't wait for the cultural zeitgeist among developers to swing back around to valuing users as human beings and living up to the trust they place in us.
Which is an elaborate way to say: you don't have to live like this.
I think it is both monopolization and cheap money. Something in economy favors big companies and disfavors competition.
Cheap money means that a selected company can run at loss, kill competition and then enshittify to start earning. By that time it is too late and too easy to buy or destroy smaller competing company. And you get access to funding by being charming to VC, by being the kind of sociopath they like to see.
Capitalism works when there is a competition. Not when there is an oligarchy.
Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.
Your description is vague enough to encompass every system that I’m aware of, from feudalism, through to mercantilism, communism, and capitalism. It may best fit communism and feudalism, both of which are most notable for fostering negative-value-added firms.
Also, though more niche, it would make archived search result pages (e.g. on the Wayback Machine) less useful.
I don't, but thanks for assuming.
Now, all Google search results appear to go back to Google.
Well, I hope ultimately this will lead to even more user loss for them. Fortunately, I myself don't have to suffer due to it, because I degooglified my life.
Google being the cesspool it is, can serve whichever (ad laden) domain that fits their own interests the best.
This is yet another move to remove power from the users.
DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.
Looking now, I've just noticed they no longer give a count of results.
For example, DST starts and ends on different dates in the UK and the US. Which dates should google return when someone from either country searches for just "DST end date?" Someone lives in Orange County and searches for "Orange County Sheriffs," which one of the eight Orange Counties should google return?
These are both examples where a localized — not even customized, just localized based on IP addresses — search results will easily help reduce headaches.
It's when two people in the same neighborhood got different results for the same search that I said "wait a minute, they're personalizing search results now for ad-targeting purposes" and ditched them. The potential for them to deliberately hide things from you was too great.
It's good for your mental health to be exposed to arguments you disagree with. Many times they will be bad arguments and you can dismiss them, but it's good to know what the people who disagree with you actually believe. And sometimes, just sometimes, you might see something that makes you realize that it's your own worldview that was mistaken, and adjust your worldview to better fit reality. If your search results only show you echo-chamber results that already agree with yours, you'll rarely learn the areas where you were mistaken. (And nearly everyone is mistaken about some things; the person who is right about everything is a rare creature indeed, and you should never gamble that you're that person).
If you could find what you want, how would they ever sell you what they want you to buy? And I'm not just talking merchandise, though that too. Your political narratives, your values, opinions, everything. And everyone likes it so much they just sit there scrolling and swiping and tapping.
Yes, the fact that they could do it does not prove that they were doing it, not right away. I ditched them as soon as I found out that they could do it, because I was absolutely certain that eventually, they would end up doing it. And I wanted neutral search results, not biased ones, even ones biased towards my own point of view.
I would love a search that lets me ban results from certain sites. I don't like search that's showing skewed results for reasons I don't know or control.
It's a lot of work to get the data to build an index. Why would they give it to everyone for free?
Similarly, Bell Labs was kind of required to release transistor for anyone to license - it was a part of social contract that they were allowed to maintain their monopoly in exchange for releasing certain parts of technology.
Alternatively, they could be split up and their indexing division made an independent company selling to anyone on a free market.
I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.
What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?
Also, there is now a use case for paid search APIs - LLMs and agents - that essentially didn’t exist a few years ago. Not sure why Google hasn’t leaned more into this, but probably a combination of Gemini and Ads interests have combined to view their search index as an increasingly valuable asset, when if anything it might be corrupted already by these interests and therefore be less valuable.
[0]: https://news.ycombinator.com/item?id=49665572
Why are Google apologists flagging my other comments?
Surely, the cost of serving search results to bots isn't that high
Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.
In effect, they’re removing attribution from the content they quote from other people’s websites. It would be interesting to see if courts object to that. It is one thing to crawl other people’s websites and display snippets of their work as your search results when each result is properly and transparently attributed. But if the text is quoted and the source is not there alongside it in plaintext, replaced only by a vague promise that, if you ask, we may or may not tell you where this piece of content is from, that is a very different deal.
It is also a measure of how enshittified and exploitative thing have become, that we look towards litigious copyright holders for assistance...
As a regular user, it's a minor annoyance.
Think about it: if you want to set up something like jina.ai you need to build your own index or piggyback on Google. My guess is they use residential proxies to fire requests to Google then scrape the URLs.
Opaque URLs now give Google another gateway to detect circumvention of their anti-bot controls, helping them monetise their search index rather than allowing providers like jina.ai to succeed.
As others have elaborated, the reasons are so they can track who you are and sell your profile advertising.
What? Like they weren’t doing this before? Obviously Google’s telemetry is tracking every link you click regardless; there’s no extra tracking benefit to this.
The reason they’re doing this seems to be to stop competitors from scraping their search results.
Brave has its own independent index which is cool.
Next i'll do is to implement tavilly, exa, tinyfish etc. as search engines for searxng. No agents, no mcp, just their search api endpoint.
Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.
And then there is the very handy assistant, that I can jump into using follow-up questions to the quick AI queries one can choose to use or not to use by appending a "?". Again a simple idea, which empowers the user.
Google's actual search results are a waste of time visiting, both for the mindless CEO content and the ad-laden, analytics-happy, javascript-heavy websites.
So I tend to use the AI overview. But plugging myself into the all-seeing corporate oracle, that grew on all the web's content, and now seeks to supplant it seems unseemly.
It's just sad that kagi will likely only be a fringe thing, and google will continue to promote these foul, foul, mindless websites and then supplant them with its AI.
- usually better than Google
- has actually useful features (smush BS listicle articles together, weight/block sites)
- if you don’t use any searches that month, they’ll just straight up refund the month subscription automatically
- doesn’t annoy me constantly
Kagi has a pile of features I’m not getting the benefit of too, I’m sure, because it’s just-search, mostly, to me.
That’s fine; I know what I’m supporting, and I know that I’m the customer and not the product.
You get the results and they do have direct URLs. But then, if you do some things with the link, e.g. right click to open it in a new tab, it swaps the URL to the indirect one. The idea is that initially you see a normal link, with a normal URL which will be displayed correctly when you hover the mouse over it, but right before you click it, it's swapped for the indirect one.
So, they have been doing stuff like this for a while and it has been somewhat fluid, because the swapping can occur on different events and I have also seen it load with all the links pre-swapped to the indirect ones, sometimes.
So, yes, what you see may be different and you may get the indirect URLs swapped at different stages.
I ran the same search in an incognito window and it showed the /goto links.
e.g. I'm just now looking for the menu for a local restaurant. "restaurantname menu" in Kagi (Google would presumably be similar) returns a link to the menu as the first result in about a second. Or "restaurantname menu !" goes directly to the menu in about a second.
Meanwhile, searching "restaurantname menu" in chatgpt takes about 5 seconds to return an embedded map from mapbox showing the location of the restaurant. If I click the restaurant pin on the map, there's no menu link, the 667 reviews have no link or way to view, and the restaurant description literally says "I don't have enough information to identify which local business <restaurantname> refers to."
Below the map there's some text: "If you mean <restaurantname> in <place>, here’s the current menu. <restaurantname>". The <restaurantname> link just opens the same card as clicking the pin on the map.
After that there's a bullet point list of the menu that ommits a ton of detail and options.
After that there's finally a link... that I can click to open up a popup at the bottom of the page with an actual link to the menu.
This was literally the first thing that popped into my head, I didn't have to put any effort into finding a query where chatgpt falls on its face.
I guess that the web chat can have a search skill to remove the prose and give only links, plus maybe an excerpt of each result
Its not because chatgpt is so superior. Its just because google search is dogshit.
They work on killing the web as we knew it and I fear its kinda working.
Streaming services already adding in ads to “ad-free” tiers they’ve now named “premium”.
Quality of life on the internet has gotten shitty while Reality Classic stays mostly the same, though more expensive.
Maybe Google is observing this?
Nothing good ever comes from businesses desperately trying to protect their moats rather than making their products better so they don't need to.
In reward for making Google Search materially worse in every way except ad revenue, he failed up again and was promoted to a cushy do-nothing role.
Everything wrong with the tech industry, embodied in a single person.
> promoted to a cushy do-nothing role
The people who this happens to are not perceived as successes. Everyone knows they are gentle firings.
Do any large ISPs use visit data to feed into a search index?
what? the Location header is the redirect, no?
Google has been encoding the target url for years.
I genuinely forget I’m not using Google until I come across articles like this
I don't miss Google at all.
Goodbye you shit company.
&udm=web works fine for me, you can use a browser shortcut or extension to enforce it
The reported behavior exists, does it not?
Anecdata: I've observed this behavior for several weeks already as a regular user without a Google account and there are countless comments from regular users reporting the same behavior.
and trying to protect your moat isn't automatically a bad thing. they clearly feel it's helping competition, so they're closing a hole. competition is good doesn't mean help your competitors.
- coming from someone who's been using fastmail as my personal for ~10 years because i don't want my emails to be backprop fodder
I’ve got something that will blow your mind. Google now has tracking analytics, for get this, your business’s phone number. Some “Adsense partner” convinced our web admin to install a little script which changes your phone number on your website so they track phone call enquires back to search engine leads / advertising spend.
Yeah no thanks, that was creepy as hell and had it rolled back ASAP. You’ve got to realise the power these tech companies hold over your business. Don’t show up in the search results, someone lists your business as closed in maps, a tracking phone number goes dead so they can’t call you, you might as well have shut up shop and ceased to exist.
Some of the things TV companies do will shock you too.
https://docs.clearurls.xyz/
I recommend reading more than the headline
What is "that" you say wouldn't work?
The extension can work fine by requesting every goto link upfront. As far as your google searches go, this is just as private as before.
I would recommend you avoid judging a comment because of a metric it neither said nor implied.
That no longer is possible, you must ask google.com for consent to visit the result link before seeing what the final target URL is.
And it can still prevent google from knowing which search results you click on, even though OP didn't mention that feature.
The article also suggests that it cannot be decoded.