Security Of Cloud Shared Links
-
Shockingly, I was misinformed about how indexing works.
I thought there was a lot more magic to it, apparently!
I now understand that standalone pages on a site cannot be indexed, except by brute force. On any site.
-
@Carnival-Boy said in Security Of Cloud Shared Links:
@BRRABill said in Security Of Cloud Shared Links:
If I don't post it to be indexed and the person I shared it with doesn't post it to be indexed, will it ever be indexed?
No, it shouldn't be. That's the purpose of YouTube's "unlisted" option. It's hidden from everyone who doesn't have the link. Search engines can't index URLs that are hidden.
That assumes that a spider can't find it. Tons of pages aren't linked anyone on any page, yet Google is aware of them because their spiders crawl all over the page doing ls commands looking for anything and everything.
Now these shared links hopefully aren't real - instead they are hopefully virtual links that tell a DB what file should be connected to, and hopefully the DB itself is not crawlable.
-
@Dashrender said in Security Of Cloud Shared Links:
That assumes that a spider can't find it.
Spidering is defined as the following of links. If it is unlinked, by definition, a spider cannot find it.
-
@StrongBad said
Spidering is defined as the following of links. If it is unlinked, by definition, a spider cannot find it.
That is what @scottalanmiller told me. (I think, don't want to put words in his mouth.)
If it's not linked, it can't be found except by brute force.
-
@Dashrender said in Security Of Cloud Shared Links:
Tons of pages aren't linked anyone on any page, yet Google is aware of them because their spiders crawl all over the page doing ls commands looking for anything and everything.
ls commands? How would they do that? There isn't any ls command in HTTP.
-
@BRRABill said in Security Of Cloud Shared Links:
@StrongBad said
Spidering is defined as the following of links. If it is unlinked, by definition, a spider cannot find it.
That is what @scottalanmiller told me. (I think, don't want to put words in his mouth.)
If it's not linked, it can't be found except by brute force.
OK I guess I used the wrong term... Google definitely knows about new pages where links to that site don't exist yet, or much - and it brute forces those sites... and it is undoubtedly brute forcing major websites looking for new pages, not waiting for links to those to appear first.
-
@Dashrender said in Security Of Cloud Shared Links:
@BRRABill said in Security Of Cloud Shared Links:
@StrongBad said
Spidering is defined as the following of links. If it is unlinked, by definition, a spider cannot find it.
That is what @scottalanmiller told me. (I think, don't want to put words in his mouth.)
If it's not linked, it can't be found except by brute force.
OK I guess I used the wrong term... Google definitely knows about new pages where links to that site don't exist yet, or much - and it brute forces those sites... and it is undoubtedly brute forcing major websites looking for new pages, not waiting for links to those to appear first.
@BRRABill and I were discussing this and this can't be possible. That would be illegal, in fact, as it would qualify as hacking. And it is technically impossible. Google and everyone else only follows published links.
-
To be sure, if you have a folder that is published or a generic name like "public" and it is listable, then you are self publishing those links through HTTP discovery, obviously. But that's publishing.
-
@StrongBad said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
Tons of pages aren't linked anyone on any page, yet Google is aware of them because their spiders crawl all over the page doing ls commands looking for anything and everything.
ls commands? How would they do that? There isn't any ls command in HTTP.
again, you're probably right, it's not ls - but there is a way to crawl over a site via HTTP - I had software 15 years ago that I just pointed toward a URL and it would find all of the folder structure that it was allowed to get to, many not having links.
-
Now we're getting to debate my question!
I originally thought the same as @Dashrender, whichis why I was concerned that the link would eventually be found.
But as you've seen, @scottalanmiller says that is impossible.
-
@scottalanmiller said in Security Of Cloud Shared Links:
To be sure, if you have a folder that is published or a generic name like "public" and it is listable, then you are self publishing those links through HTTP discovery, obviously. But that's publishing.
This is what I'm talking about.
-
@Dashrender said in Security Of Cloud Shared Links:
@StrongBad said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
Tons of pages aren't linked anyone on any page, yet Google is aware of them because their spiders crawl all over the page doing ls commands looking for anything and everything.
ls commands? How would they do that? There isn't any ls command in HTTP.
again, you're probably right, it's not ls - but there is a way to crawl over a site via HTTP - I had software 15 years ago that I just pointed toward a URL and it would find all of the folder structure that it was allowed to get to, many not having links.
The folders present links via HTTP. Those are linked. Nothing nefarious or weird there, that's a published directory structure. You can do it by hand and see the links very clearly.
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
To be sure, if you have a folder that is published or a generic name like "public" and it is listable, then you are self publishing those links through HTTP discovery, obviously. But that's publishing.
This is what I'm talking about.
Right, so if there are links, Google can see them. Disable the display of the links, and Google cannot.
-
@BRRABill said in Security Of Cloud Shared Links:
Now we're getting to debate my question!
I originally thought the same as @Dashrender, whichis why I was concerned that the link would eventually be found.
But as you've seen, @scottalanmiller says that is impossible.
But he doesn't - he and I are talking about the same thing - things that you self publish to HTTP are there and are findable without links from some place else.
-
@Dashrender said in Security Of Cloud Shared Links:
@BRRABill said in Security Of Cloud Shared Links:
Now we're getting to debate my question!
I originally thought the same as @Dashrender, whichis why I was concerned that the link would eventually be found.
But as you've seen, @scottalanmiller says that is impossible.
But he doesn't - he and I are talking about the same thing - things that you self publish to HTTP are there and are findable without links from some place else.
No one said links from somewhere else. You are linking yourself to every file in the example that you are providing.
-
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
To be sure, if you have a folder that is published or a generic name like "public" and it is listable, then you are self publishing those links through HTTP discovery, obviously. But that's publishing.
This is what I'm talking about.
Right, so if there are links, Google can see them. Disable the display of the links, and Google cannot.
I'm not entirely sure what you mean by display of links - can you be more specific in an explanation?
-
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@BRRABill said in Security Of Cloud Shared Links:
Now we're getting to debate my question!
I originally thought the same as @Dashrender, whichis why I was concerned that the link would eventually be found.
But as you've seen, @scottalanmiller says that is impossible.
But he doesn't - he and I are talking about the same thing - things that you self publish to HTTP are there and are findable without links from some place else.
No one said links from somewhere else. You are linking yourself to every file in the example that you are providing.
So you're saying that every file in the www root directory on an IIS server is considered self published or more specifically.. self linked? Even if there is no link from any htm page that is on the site?
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
To be sure, if you have a folder that is published or a generic name like "public" and it is listable, then you are self publishing those links through HTTP discovery, obviously. But that's publishing.
This is what I'm talking about.
Right, so if there are links, Google can see them. Disable the display of the links, and Google cannot.
I'm not entirely sure what you mean by display of links - can you be more specific in an explanation?
We rarely see this today because no one does this, we use applications rather than straight files, but let's say you have a directory of HTML files under my.site.com/files/
You can set the web server to automatically generate a page as the default for that folder that displays each file in that folder as a link. This is not part of the web or of HTTP, but is a function that can be enabled in some web servers (but not all.) It's a "auto linking" feature that people often want. But the web server does this explicitly and makes a link to each resources creating everything that spiders need to see all files, including link to the next directory listing.
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@BRRABill said in Security Of Cloud Shared Links:
Now we're getting to debate my question!
I originally thought the same as @Dashrender, whichis why I was concerned that the link would eventually be found.
But as you've seen, @scottalanmiller says that is impossible.
But he doesn't - he and I are talking about the same thing - things that you self publish to HTTP are there and are findable without links from some place else.
No one said links from somewhere else. You are linking yourself to every file in the example that you are providing.
So you're saying that every file in the www root directory on an IIS server is considered self published or more specifically.. self linked? Even if there is no link from any htm page that is on the site?
No, but you don't have that happening in your example. In your example, you are creating links to those resources. Most people do, as they want them spidered.
-
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
