New Cloud Storage Feature: Creating File Download Links

storage-sharing

So, a new feature has appeared in our cloud storage: creating download links. By creating such a link, you can send it to any third-party user so that he uploads his files to you - completely free. It is not even necessary for this user to have an account in the repository.

This function may be useful for solving a variety of tasks, for example:
  • to provide any user access to your storage: for example, friends will be able to upload their photos, audio and video recordings for you, colleagues and partners - various working materials, etc.
  • to upload custom content to sites;
  • To simplify file sharing between members of thematic online communities;
  • for quick backup backups.


Background


I recently participated in climbing Mount Elbrus. All members of the group, including mine, had cameras with them. Many interesting and beautiful photographs were taken, and after the trip the question arose: how to make them accessible to all participants?

Someone suggested uploading all the pictures on VKontakte or on Facebook, someone on Mail.Ru, someone on other file hosting ... But all of the above services did not suit us: firstly, there are restrictions on the size of the downloaded files; secondly, not all members of the group were registered on these services.

I proposed a simple and convenient solution for everyone: I created a public container in Selectel cloud storage, turned on the option for listing files. Then he created a link, by clicking on which everyone could upload their own photos to the container, as well as download photos taken by other participants. At the same time, none of the group members needed to create an account in the repository. So a new feature of our cloud storage was tested - creating a link for downloading files by third-party users. My climbers on Mount Elbrus rated it "excellent."

Today we introduce this feature to a wide audience. We hope that all of our users will appreciate it.

Link creation


Let's look at a simple example of how a new function works. Create a special container for downloading files and name it, for example, share. Then open the container menu and select the “Link to download files” item:
The link settings window opens:
In this dialog box you can set the following parameters (more on some of them will be discussed below):
  • choose the type of link;
  • enter explanatory text for the users to whom you send the download link (the explanatory text will be displayed in the subtitle at the top of the download page);
  • set a password for downloading files;
  • set limits on the maximum size of files uploaded to the repository;
  • limit the duration of the link, as well as the retention period of the downloaded files;
  • enable / disable file overwriting when uploading files with the same name to the repository (if this option is enabled, the old file will be deleted and the new one will be saved; if disabled, the old file will not be deleted, and an additional identifier will be added to the name of the new file).

Now click on the "Create Link" button. A dialog box appears with a link that you can copy and send to any other user. For each container or folder, you can create an unlimited number of download links with various combinations of settings.

Download Page


The user who received the link from you and opened it in the browser will see the following page:
The page supports simultaneous parallel downloading of three files, and due to this, the download process is significantly accelerated:
If any files need to be deleted, this can be done within 10 minutes after the download is complete.

Link Settings


Types of Links


As mentioned above, when creating a link, you need to select its type. We have provided several types of links adapted for different use cases. Let's consider each type of links in more detail.

Easy download


Suitable for situations where you need to receive files from one or more users. Files will be uploaded to the specified container or folder, and you can access them both through the web panel or in other ways.

Adding Download Times to a File Name


If you select this option, the date and time of the download will be added to the name of the downloaded file. For example, if you upload the file myphoto.jpg, it will be saved as myphoto_20130830_12h43m53s.jpg.
This can be useful when saving backups, as well as in cases where access to the download link is provided to a large number of users, and it is necessary to exclude matches in the names of the downloaded files.

Adding a unique identifier to the file name


If you select this option, a unique identifier will be added to the file name. For example, the file myphoto.jpg after downloading will be renamed to myphoto_cmrsm2ks.jpg.
This type of link is suitable when uploading user-generated content to sites (in this case, it’s also useful to turn on the option to ignore the original file name): the name given by the user will be rejected, and the new file name will correspond to its unique identifier.

Download to a unique folder


If you select this type of link, the files during each download session will be saved in a separate folder. For example, the file myphoto.jpg will be saved in a folder named cmrsm2ks. If after downloading this file you close the download page and then close it again and load other files, they will be saved in another folder, the name of which will be generated automatically; a separate folder will also be created for the next download session, etc.

Upload to folder by date


This mode of operation is well suited for downloading backups. The downloaded files will be saved in the folder with the current date in the name. For example, the mysql.dump.bz2 archive after downloading will be available as 2013-08-30 / mysql.dump.bz2.

Upload to the folder by the clock


This mode is generally similar to the previous one; the only difference is that in addition to the date, the download time (hour) will be added to the folder name. The mysql.dump.bz2 archive will be available as 2013-08-30_15 / mysql.dump.bz2.

Download with the specified name


This type of link is more suitable for use through the API for applications or sites.
The link will be valid for only one download, and the downloaded file will be renamed by the name of the link itself. For example, if the link looked like xxx.selcdn.ru/container/you_photo.jpg , then after downloading the file will be available in the repository under the name you_photo.jpg, and the link will stop working.

Temporary restrictions


When creating a download link, you can set time limits:
This is a very flexible and useful option. It allows, firstly, to limit the duration of the link itself. After the period you specified, third-party users will not be able to upload files to the repository. This feature may be useful in situations where it is necessary to receive documents from users via the Internet by a certain date (for example, when accepting applications for conferences and competitions).

Secondly, it can be used to set the retention period of downloaded files. You can set a specific period of time during which files from the user who received the download link from you will be stored. At the end of this period, all files will be automatically deleted. Such functionality will be useful, for example, for organizing the loading and storage of backups.

Additional features


For developers


Link management can be done through the API and is compatible with the Openstack Swift API. Using libraries for working with the Openstack Swift API, you can easily create and configure links for uploading files to our repository directly from your own applications and sites.

Using the download links, you can implement the ability to download content from a third-party site to our repository.
Here is a simple example:
File upload form:
<form id="put-file">
 <input id="fileToUpload" type="file" />
 <input type="submit" />
</form>


Javascript code:
(function() {
 var $putFile = document.getElementById('put-file');

 $putFile.onsubmit = function (event) {
 var req = new XMLHttsourcequest();
 event.sourceventDefault()

 file = document.getElementById('fileToUpload').files[0]

 req.open("PUT", "https://xxx.selcdn.ru/containet/sendmefile_link");
 req.withCredentials = false;
 /* исходное имя файла */
 req.setRequestHeader("X-Filename", file.name);
 /* небольшой хак для установки правильного времени для загружаемого файла */
 TZ_OFFSET = (new Date).getTimezoneOffset();
 req.setRequestHeader("X-Sendmefile-TZ-Offset", TZ_OFFSET);
 req.send(file);
 req.onreadystatechange = function () {
 if(req.readyState === 4 && req.status === 200)
 alert(req.responseText)
 }

 return false;
 }
})()


For system administrators


The function of creating download links can be useful for quickly downloading files from the server to the storage, as well as for setting up backups.

To download a file, you must first create a link in the web panel, and then run the following command in the server management console using the curl utility:

$ curl -i https://xxx.selcdn.ru/container/upload_link -T path_to/file.name -H "X-Filename: file.name"

You can also organize the transfer of data through a named pipe (pipe), without saving to a file:

# загрузка со сжатием в архив "на лету"
$ bzip2 bigfile.dat -c | curl -i http://xxx.selcdn.ru/container/upload_link -X PUT -d @- -H "X-Filename: bigfile.dat.bz2"

Another interesting feature is multi-threaded downloading of files from a specific folder; it will be much faster than using our supload utility :

# паралельность определяется опцией -P
$ find path_to_dir/ -type f | xargs -I "{}" -n1 -P 10 curl http://xxx.selcdn.ru/container/upload_link -i -X PUT -H "X-Filename: {}" -T "{}"


Backup


The new function greatly simplifies the backup procedure: now it is not necessary to create separate containers for each server or site; no need to create additional users whose number is limited. To ensure that backups are downloaded and stored, you just need to create a password-protected link and configure it accordingly.

Creating links to download backups allows you to solve a number of problems that previously required considerable effort.

The first is to sort downloaded backups by date and time. To ensure that all downloaded files are automatically sorted or renamed according to the date and time, just create a link of the appropriate type.

Secondly, it automatically deletes backups after the expiration of the retention period. The storage period can be set in two ways: set a strictly specified time and the default time. The second option allows you to change the shelf life during the download process.

Backup via download links can be easily integrated into automatic backup systems, which very often support the execution of an external command after creating an archive with a copy of the data.

Afterword


As often happens with us, a new function became available to the user a few days before it was officially announced. Some of our clients have already tried to “tie” it to their own services, and we have already received feedback from them. In their reviews, they expressed us a number of comments and suggestions that will be taken into account in the very near future. Very interesting wishes for the future were expressed, in particular:
  • implement the ability to generate links for downloading files uploaded to the repository;
  • create hashed passwords, which allows you to not create a new password-protected link each time you download user-generated content from sites.

We invite all our readers to try the new feature. Link to the test version of the download page: http://23728.selcdn.ru/sendmefile-test/upload (all files will be stored for an hour, the link itself will be available 30 days from the date of publication of the article).

We will be glad to any comments, suggestions, suggestions and be sure to take them into account in future work.
For those who cannot comment on posts on Habré, we invite to our blog .