Security Of Cloud Shared Links
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@BRRABill said in Security Of Cloud Shared Links:
Now we're getting to debate my question!
I originally thought the same as @Dashrender, whichis why I was concerned that the link would eventually be found.
But as you've seen, @scottalanmiller says that is impossible.
But he doesn't - he and I are talking about the same thing - things that you self publish to HTTP are there and are findable without links from some place else.
No one said links from somewhere else. You are linking yourself to every file in the example that you are providing.
So you're saying that every file in the www root directory on an IIS server is considered self published or more specifically.. self linked? Even if there is no link from any htm page that is on the site?
No, but you don't have that happening in your example. In your example, you are creating links to those resources. Most people do, as they want them spidered.
-
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
-
Example...
http://mirror.centos.org/centos/
That link doesn't go to a file, it goes to a directory. but what does your browser display? An HTML page automatically generated by the web server that links to all files and folders stored in that location. Follow the links to automatically generated pages with more links until you get to the file that you want. A spider following this is literally following links generated by an application making HTML pages behind the scenes to display the links to the end users (or spiders.) This is not intrinsic but is a "by convention" method of displaying static HTML folders and files and is super common to not use (the web server that NodeBB uses doesn't even have this functionality.)
If you don't automatically make those links one way or another, the spider has nothing to follow.
-
Things that people often miss that make links that they don't know about are...
- Automatic links generated for default landing by the web server
- Automatic links generated for folder listing by the web server
- Sitemaps generated by the web server or application
- Robot directives directing spiders to specific resources
- RSS or Atom feeds of files
- Automatic linking by applications
-
@scottalanmiller said in Security Of Cloud Shared Links:
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
auto linked to what?
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
auto linked to what?
To whatever the default is set in the web server. Go to http://ntg.co/ (really, go to it, we need the hits) and the web server is informed to go "serve up the default link." In the case of that particular site, the default is set to index.php. You set this for all web servers. If you don't set it for Apache, it has a default setting of index.html and IIS has a built in default of index.htm.
-
@scottalanmiller said in Security Of Cloud Shared Links:
Example...
http://mirror.centos.org/centos/
That link doesn't go to a file, it goes to a directory. but what does your browser display? An HTML page automatically generated by the web server that links to all files and folders stored in that location. Follow the links to automatically generated pages with more links until you get to the file that you want. A spider following this is literally following links generated by an application making HTML pages behind the scenes to display the links to the end users (or spiders.) This is not intrinsic but is a "by convention" method of displaying static HTML folders and files and is super common to not use (the web server that NodeBB uses doesn't even have this functionality.)
If you don't automatically make those links one way or another, the spider has nothing to follow.
OK that makes sense. So if, in the case of NodeBB, it doesn't have the functionality, but there a folder called /pictures123 that has pictures in it and there are no links to it, and not auto generation - yet if you know the exact URL, you're saying Google can't find that folder? and no legal entity can?
-
Right... the HTTP command set has no listing capability at all. That's not one of the HTTP commands. All directory listings or links of any sort have to be either included in a static file or created by the web server or something that talks to the web server (like a PHP site.)
-
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
auto linked to what?
To whatever the default is set in the web server. Go to http://ntg.co/ (really, go to it, we need the hits) and the web server is informed to go "serve up the default link." In the case of that particular site, the default is set to index.php. You set this for all web servers. If you don't set it for Apache, it has a default setting of index.html and IIS has a built in default of index.htm.
The whole it's linked is where I'm getting hung up here - I agree that those are the default locations where those services will send people - but calling them links or linked - not sure you've convinced me yet.
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
Example...
http://mirror.centos.org/centos/
That link doesn't go to a file, it goes to a directory. but what does your browser display? An HTML page automatically generated by the web server that links to all files and folders stored in that location. Follow the links to automatically generated pages with more links until you get to the file that you want. A spider following this is literally following links generated by an application making HTML pages behind the scenes to display the links to the end users (or spiders.) This is not intrinsic but is a "by convention" method of displaying static HTML folders and files and is super common to not use (the web server that NodeBB uses doesn't even have this functionality.)
If you don't automatically make those links one way or another, the spider has nothing to follow.
OK that makes sense. So if, in the case of NodeBB, it doesn't have the functionality, but there a folder called /pictures123 that has pictures in it and there are no links to it, and not auto generation - yet if you know the exact URL, you're saying Google can't find that folder? and no legal entity can?
Right, Google has no means to look for that folder or its contents.
-
@StrongBad said in Security Of Cloud Shared Links:
Right... the HTTP command set has no listing capability at all. That's not one of the HTTP commands. All directory listings or links of any sort have to be either included in a static file or created by the web server or something that talks to the web server (like a PHP site.)
OK, that was something I didn't know. Thanks.
Then what's the problem with leaving the default directories and junk behind in an IIS install - if nothing links to those things, what harm is there in them being there? I suppose their being there as a directly accessible folder and the tyranny of the default is what creates the harm. So if you know that that is a default folder, you can try to go there directly and attempt to execute something that might be there...
-
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
auto linked to what?
To whatever the default is set in the web server. Go to http://ntg.co/ (really, go to it, we need the hits) and the web server is informed to go "serve up the default link." In the case of that particular site, the default is set to index.php. You set this for all web servers. If you don't set it for Apache, it has a default setting of index.html and IIS has a built in default of index.htm.
The whole it's linked is where I'm getting hung up here - I agree that those are the default locations where those services will send people - but calling them links or linked - not sure you've convinced me yet.
It only sends people there because of the links. You can argue that index.html is not a link, but only that one case. All other resources are only available by links, there is no default to get people there.
-
@Dashrender said in Security Of Cloud Shared Links:
Then what's the problem with leaving the default directories and junk behind in an IIS install - if nothing links to those things, what harm is there in them being there? I suppose their being there as a directly accessible folder and the tyranny of the default is what creates the harm. So if you know that that is a default folder, you can try to go there directly and attempt to execute something that might be there...
Because if it is generic, then malicious users can try to access it because they know that it is commonly there. It's a bigger attack surface.
-
My example I used was that if I create a page OUTSIDE my WordPress site, there is no way anything can find it.
So if I made www.brra.com/SAM, no one would ever see it.
Well, they would now because I posted here on ML. But you get the drift.
-
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
@Dashrender said in Security Of Cloud Shared Links:
@scottalanmiller said in Security Of Cloud Shared Links:
Even index.html isn't an exception to this, that is autolinked as a setting in the web server, too. Many of the links are by convention.
auto linked to what?
To whatever the default is set in the web server. Go to http://ntg.co/ (really, go to it, we need the hits) and the web server is informed to go "serve up the default link." In the case of that particular site, the default is set to index.php. You set this for all web servers. If you don't set it for Apache, it has a default setting of index.html and IIS has a built in default of index.htm.
The whole it's linked is where I'm getting hung up here - I agree that those are the default locations where those services will send people - but calling them links or linked - not sure you've convinced me yet.
It only sends people there because of the links. You can argue that index.html is not a link, but only that one case. All other resources are only available by links, there is no default to get people there.
I completely agree with the rest being links.. just the default page loaded when visiting a folder directly be it www.google.com or www.google.com\scottalenmiller
-
@BRRABill said in Security Of Cloud Shared Links:
My example I used was that if I create a page OUTSIDE my WordPress site, there is no way anything can find it.
So if I made www.brra.com/SAM, no one would ever see it.
Well, they would now because I posted here on ML. But you get the drift.
Yesterday I would have said that Google could find it.. but now with an education from Scott and strongbad - I guess not.
-
@Dashrender said
Yesterday I would have said that Google could find it.. but now with an education from Scott and strongbad - I guess not.
You and me, both.
I really thought that is how it worked. It crawled through the SITE looking for files, not looking for links.
-
@BRRABill said in Security Of Cloud Shared Links:
@Dashrender said
Yesterday I would have said that Google could find it.. but now with an education from Scott and strongbad - I guess not.
You and me, both.
I really thought that is how it worked. It crawled through the SITE looking for files, not looking for links.
Well - that's kinda semantics, but not entirely.
What I didn't know, that @StrongBad pointed out, is that the HTTP protocol has not way of displaying content of a folder itself. That those webservers that do show the folder contents do so because of a function of the web server, not a function of HTTP - and on the web server side, it can be turned off - which was something I know could happen, but I didn't know to what level it actually kept people out - sounds like it actually does a pretty damned good job. -
I didn't realize I could put items outside the realm on my site and not have them seen.
Sweet.
-
Then there's the otherside of this - the fact that there aren't that many static pages anymore. Most of the time things are generated on the fly by an application installed into the web server, such as WordPress. So even if you could search the directory, there wouldn't be anything there. instead the file is created only upon request and delivered to the end user, and not written to the directory.
