Timeout in .sh file

scottalanmiller

@thwr

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@stacksofplates said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

I would always use curl / wget / any other CLI tool before I would use Chromium (Chromium != Chrome) for a simple file download. It's a monster in terms of resource consumption.

I tried wget where credentials is asking and in curl the file is not downloading where the test file created inside that file the link is there to download the file.So only using Chrome here

Show us the command you ran. I have a feeling you’re either not following a redirect or your output option is not set correctly.

curl -o "Link"

Could you please try what I posted above? And could you please bring some light in your requirements? What kind of authentication is required by the server? Basic HTTP Auth?

The process here is the internal site is there where the .csv files will be downloaded from the site through wget or curl or by browser.Where i tried from the wget i got the error as

"HTTP request sent, awaiting response... 401 Unauthorized"-->I think this for the AD username and password where the credentials part is not even asking.

How to try the above thing which was mentioned already

So how does the site work, it just has a normal login form?

Lakshmana

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

black3dynamite

@scottalanmiller @stacksofplates @thwr

Is there a way to prompt the user for the credentials instead?

scottalanmiller

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

Lakshmana

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

thwr

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

Yep. And it's a very bad idea to rely on something you can't control: a users cached cookies.

scottalanmiller

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

Right, so that is a reason to avoid that. you need to get the cookie cache and manage it. It's specifically for this kind of reason that cURL is the right tool. I've done exactly this in the past for automation where we needed a login AND a cookie cache and cURL handles this very well.

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

Yep. And it's a very bad idea to rely on something you can't control: a users cached cookies.

This script will not run on users .only on System Administrator machines only

thwr

That doesn't matter at all. There are only a few factors:

Do you need any form of authentication - if so, what exactly? A web based form, a HTTP BASIC AUTH (your browser shows a popup for credentials) or something else (some auth protocols can be tricky)
Can the machine running the script reach the webserver?
Is there any script language available? not required, just a plus
Any download tool available? curl, powershell?

You see, it's pretty simple. Most questions are already answered. We now need to know what kind of authentication your webserver requires. Simply start your favorite browser in incognito mode (shift + ctrl + p in Chrome) and navigate to your URL. There are no cookies or caches involved in that moment. Tell us what you see, maybe create a screenshot.

Lakshmana

@thwr said in Timeout in .sh file:

That doesn't matter at all. There are only a few factors:

Do you need any form of authentication - if so, what exactly? A web based form, a HTTP BASIC AUTH (your browser shows a popup for credentials) or something else (some auth protocols can be tricky)

Can the machine running the script reach the webserver?

Is there any script language available? not required, just a plus

Any download tool available? curl, powershell?

You see, it's pretty simple. Most questions are already answered. We now need to know what kind of authentication your webserver requires. Simply start your favorite browser in incognito mode (shift + ctrl + p in Chrome) and navigate to your URL. There are no cookies or caches involved in that moment. Tell us what you see, maybe create a screenshot.

As checked the link in the browser by incognito window there is not Login Credentials was asked where the file get downloaded without any issues.

1)As of now i dont need any authorization to download the file
2)The machine which is running is able to reach the Webserver
3)No script language.Just making .sh file to download the files consecutively
4)I am trying the process in Ubuntu and same needs to be worked in Mac but tried the Curl which is creating the file which have the same link inside that file once i click on that link the file will be downloading

I need to download the Files automatically in Downloads Test Folder in the .csv format first

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

That doesn't matter at all. There are only a few factors:

Do you need any form of authentication - if so, what exactly? A web based form, a HTTP BASIC AUTH (your browser shows a popup for credentials) or something else (some auth protocols can be tricky)

Can the machine running the script reach the webserver?

Is there any script language available? not required, just a plus

Any download tool available? curl, powershell?

You see, it's pretty simple. Most questions are already answered. We now need to know what kind of authentication your webserver requires. Simply start your favorite browser in incognito mode (shift + ctrl + p in Chrome) and navigate to your URL. There are no cookies or caches involved in that moment. Tell us what you see, maybe create a screenshot.

As checked the link in the browser by incognito window there is not Login Credentials was asked where the file get downloaded without any issues.

1)As of now i dont need any authorization to download the file
2)The machine which is running is able to reach the Webserver
3)No script language.Just making .sh file to download the files consecutively
4)I am trying the process in Ubuntu and same needs to be worked in Mac but tried the Curl which is creating the file which have the same link inside that file once i click on that link the file will be downloading

I need to download the Files automatically in Downloads Test Folder in the .csv format first

Ah, very good. So we can safely ignore any authentication. Mostly. It's still possible that there's some kind of authentication you can't see this way, like IP source or user agent checks for example. A user agent string is basically a way to tell a webserver which browser is asking for files. BTW: sh / bash is a language on its own But let's skip that for now.

So what exactly happens when you just enter

wget -O /home/user/download/file1.csv https://your.webserver/folder/file1.csv

Is there any error message?

Lakshmana

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

thwr

@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:

curl -JLO https://your.webserver/folder/file1.csv

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv

The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office

Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?

The .csv file is there but the file is having another link to download the original .csv file

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?
The .csv file is there but the file is having another link to download the original .csv file

what do you mean by "another link"? Is the link the content of the file you've downloaded?

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?
The .csv file is there but the file is having another link to download the original .csv file
what do you mean by "another link"? Is the link the content of the file you've downloaded?

The example is "facebook,com/example.mp4" that example.mp4 is the file which needs to be downloaded but the file download as example and inside that example file Click here Link(facebook.com/example.mp4) is present and i need to click that to download that file