Auto WGet Daemon

Auto WGet Daemon Great idea and first release (C) 1998 Steve Trubachev Final release (C) 1998-2005 Dmitry A.Steklenev CONTENTS: Short description Download utilities Installation Click catching modules installation Important notice How it works How to feed necessary URL to daemon Using in the network Some words about NFTP Netscape and Mozilla click catching modules Configuration Known bugs Additional programs Plugins Job file About Special thanks Short description Auto WGet Daemon is a free utility set for driving GNU WGet or any similar utility for downloading files. If you had been tired after you found necessary link in Netscape Navigator, copy it to clipboard and start WGet by hands (God! Are you so lazy? :), Auto WGet Daemon is that thing, you need. Auto WGet Daemon allows you to start downloading by simply Drag'n'Drop to special folder. Aside from, Auto WGet Daemon supports queue for downloads and allows restart failed downloads in the power fail or any other unpleasantness. Download utilities At this time only two utilities are supported: GNU WGet and cURL. For a normal functionality Auto WGet Daemon you need to install either. Also you need make sure they are working. They must to be placed to any directory in PATH statement in CONFIG.SYS file. OS/2 version of GNU WGet you can find at Hobbes: http://http://hobbes.nmsu.edu/ (search "wget") cUrl for OS/2 you can find at support homesite: http://curl.haxx.se Installation You need unpack archive file to any temporary folder and start INSTALL.CMD, to response to a couple of questions and all must be OK. Installation program will create special folders named "ToDO" and "Auto WGet daemon" at your Desktop. In addition, in %ETC% will be created (or updated, if it isn't first installation) configuration file AWGET.CFG. Object for daemon starting will be placed to "Startup" folder. You don't need restart computer after installation. Daemon will start immediately after installation. For normal functional Auto WGet Daemon must be installed to any long-name and extended attributes supported filesystem (HPFS, HPFS386, JFS or FAT32). Click catching modules installation Starting from Auto WGet Daemon version 1.9.0 Netscape and Mozilla click catching modules are included (refer to part "Netscape and Mozilla click catching modules" for details). Installation of the modules is performed automatically while Auto WGet Daemon is being installed and by user request then. But a situation while installer fails to find all supported browsers properly is possible. In such case, it is recommended to copy modules manualy. In Auto WGet Daemon installation package you can find 3 modules for different file types: NPAWGBIN.DLL - for binary files NPAWGAUD.DLL - for audio files NPAWGVID.DLL - for video files Copy desired modules into "Plugins" folder in case of Mozilla or "Program/Plugins" for Netscape. After browser restart modules will be loaded and ready for use. Calling "Help->About Plug-ins" you can check modules installation successfull completion. Important notice This version of Auto WGet Daemon can be installed over previous version less than 1.6.1, but this isn't recommended strongly. If you are using version like this, you need de-install it, delete %ETC%\AWGET.CFG file and only after that install new one. How it works Auto WGet daemon scans "ToDo" folder and sub-folder contents continuously. As soon as object with URL appears in those folders, download utility starts. At that object will be moved to Active folder. It is signal that download process had been started. In case of broken download or computer restarts all objects from Active folder will be moved to previous place and will be proceed in the regular manner. How to feed necessary URL to daemon First method: when you see necessary URL, you can drag it by mouse to "ToDo" folder (or subfolder). This URL will be placed to downloads queue. Second method: you need click right mouse button at the URL in the browser window and select "Copy Link Location". After that you need click right mouse button at "ToDo" folder icon or subfolder and select "Paste...". You can or can not enter other name for object. It doesn't matter. After all you need press "Paste". Third method: right click on the URL in the browser window and choose "Copy Link Location". After that, start "Add URL" from the "Tools & Utilities" folder of Auto WGet Daemon, paste URL to the edit-box if it doesn't appear there (by pressing SHIFT-INS or mouse buttons simultaneously) and click on the "Add". By default object with appropriate URL will be created in the "ToDo" folder, but you can choose different sub-folder using combo-box over edit-box. Fourth method: you need prepare ASCII file with URL list (as sample NFTP does this), execute "Add URL" and put to the edit box filename with list. In the "ToDo" folder will be created group of URLs objects from list. Fifth method: you need set USE_DESKTOP in configuration file to "1". After this you will able to put URL by first and second method to Desktop directly. Daemon will move it to "ToDo" folder. Sixth method: set CLIPBOARD_MONITOR key value in configuration file to 1 and do everything the same as described in the third method with one exclusion only - add URL window will come up automatically after copying URL to clipboard is done. Using in the network You can share "ToDo" and download folder and use server with installed Auto WGet Daemon as common download center either as from OS/2 as from Windows. In the case you have to drop URL in the folder. You can also establish client-server relations between two computers working under OS/2 where Auto WGet Daemon is installed. Next example with two computers having NetBIOS names SERVER and CLIENT respectively will show you how to do it. On SERVER you have to create subfolder with name at your will in "ToDo" folder. For an instance we call it "FromClient". Drag additional configuration template file awget.cfg (located in "Tools & Utilities") into this folder and edit it. You have to uncomment and change values for the following parameters: MESSAGE_DONE = net send CLIENT "%M: %U" MESSAGE_ERROR = net send CLIENT "%M: %U" Additionally you have to assign to CLIENT reading and writing permissions to the folder. In our case we share folder where Auto WGet Daemon is installed. Now we create subfolder "ToServer" in "ToDo" folder on CLIENT and force download redirection for the folder in additional configuration file: REDIRECT_TO = \\server\awget\folders\todo\fromclient That is it. Each URL dragged into "ToServer" folder on CLIENT will be moved to "FromClient" folder on SERVER side and download completion messages will be sent back to CLIENT by SERVER. Notice: creating special subfolder on CLIENT is not a must, you may just put awget.cfg into "ToDo" folder. Some words about NFTP NFTP is well known FTP client for OS/2. It is being developed by Sergey Ayukov. It can be found at http://www.ayukov.com. To use Auto WGet Daemon with NFTP it is enough to alternate two lines only in nftp.ini (don't forget to remove comment symbol ";" before them) launch-wget = "x:\path_to_awget\awgadd.cmd %s" enable-passworded-wget = yes Where "x:\path_to_awget" needs to be replaced to the directory where Auto WGet Daemon is located. Having this done you may issue download request to daemon highlighting desired files and pressing Shift-W. Netscape and Mozilla click catching modules Netscape and Mozilla click catching modules are designated for defined type links interception for further processing by the daemon. Because of a browser pluggable in modules design peculiarity, functionality of provided catchers is limited: - Impossible to change list for file types being downloaded; - In case of refusal to transfer the file for Auto WGet Daemon processing, the file will not be proceed by browser anyway; - Impossible to download a file from a password protected server (excluding a case when user name and password are mentioned in the link directly). To provide at least minimum of the modules configuration ability, there are three modules included in the installation package for different file types which installation you may specify while daemon is being installed. Also if there are problems while file downloading using Auto WGet Daemon, try click on a the link keeping SHIFT key pressed. If will force a browser to start file download. Configuration Main configuration file (named "AWGET.CFG") is lying in the %ETC% directory (usually point to \MPTN\ETC). There is an object in the "Tools & Utilities" folder for editing configuration file. Statement description is written in the configuration file body as comments for each. You can create subfolders in "ToDo" folder and put in their additional configuration files. Use this future you can change some statements from main configuration file (folder processing time, download directory, etc.) A template for creating file like this can be found in "Tools & Utilities" folder. Don't forget erase comment begins symbol "#" before statement you want to change. Known bugs - Auto WGet Daemon can't understand URL objects created by Netscape Communicator 4.61 in OS/2 Warp 3.0. You can use Netscape Navigator 2.02, Netscape Communicator 4.04 or send the URL by "Add URL" object. Or you can use OS/2 Warp 4.0 and better :) - This software is incompatible with Vadim Yegorov aka VYX. Additional programs Pmpopup2.exe Ulrich Sturmberg PM Popup Version 1.0 Aside from, you need netstat utility from standard TCP/IP in check for PPP connection case. Plugins Plugins are programs for improve and extend functionality of Auto WGet Daemon. Two types of modules are supported - global and private. Global modules interact with daemon directly and are especially oriented to affect to job planning process. Private modules interact with each of started download processes and can affect at it execution manner. To plug a module in you need add description to main configuration file AWGET.CFG. Use the key GLOBAL_PLUGIN for global modules and PRIVATE_PLUGIN for private. A sample: GLOBAL_PLUGIN = Plugins\awpglob.cmd For activating a new global module you need restart daemon. Private module will be plugged in as soon as particular download process will be started. Here is the info for the developers. Pluggable module is a detached process. It is running with daemon (global modules) or with each particular download process (private modules). You can interact with module through standard input/output streams. Communication accomplishes in string by string mode using special commands. Each string contains only one command. After posting string to module, answer must be received and daemon execution will be stopped until answer appears. If configuration file contains more than one module, all modules will be served in order of definition. Command contains four symbol size keyword and optional information part. Between command and information part one space symbol must be inserted. Only two answers from information module are allowed: DONE and FAIL. DONE denotes successful command execution. FAIL denotes existing of error or execution restriction for the function asked. The information sent with DONE, is printed to the screen. The information sent with FAIL is printed in the error log and to screen. Module can send one or more commands for additional info (INFO, EVNT and ALRM) before sending DONE or FAIL. Info, following any of those commands, will be printed at the screen. Also, info from EVNT command will be recorded to event log and ALRM to error log. To interact with global modules next commands can be used: INIT awget_home_directory Will be sent in initialization time of Auto WGet Daemon. Module answer will be ignored. SCAN todo_directory Will be sent before scanning of Desktop and "ToDo" folder begins. In the case of FAIL answer at least of one module, scanning will be passed through. CONF configuration_file Will be sent after indicating of main configuration file changes. Module answer will be ignored. SEXE job_file Will be sent before starting download process. Module can change job file content. In the case of FAIL answer, startup will postpone. SEND job_file Will be sent after download process completion. Module can change job file content. Module answer will ignore. STOP Will send before daemon's shutdown. Daemon won't wait for an answer from module. It will shutdown immediately after sending command to all modules. To interact with private modules next commands can be used: INIT job_file Will be sent in the download initialization moment. Module can change job file content. In the case of FAIL answer at least of one module, process will stop. DATA Will be sent before download utility output translation begins. After answer DONE, module must change input mode to symbol. Translation will end by '00'x. STOP job_file Will be sent before download process shutdown. Module can change job file content. Module answer will ignore. Samples for global and private plugin can be found in the "Plugins" directory of Auto WGet daemon. Job file All the information required by the download process can be sent through job file. Job file format is like configuration file and contains pairs - key and value. Next keys are used: OBJECT Object with URL name, which will proceed. URL Download URL. DOWNLOAD Folder, where file will placed after download. MESSAGE_DONE Successful download command. MESSAGE_ERROR Broken download command. DOWNLOADS_UTILITY Download utility. DOWNLOADS_PARAMETERS Download utility parameters. DOWNLOADS_RC Download utility exit code. DOWNLOADS_INFO Download utility last message. Some samples for reading and writing job files can be found in awpglob.cmd and awppriv.cmd files in "Plugins" directory. About Great idea and first release (C) 1998 Steve Trubachev Final release (C) 1998-2005 Dmitry A.Steklenev You can send comments to: dmitry@5nets.ru Auto WGet Daemon is free software; this means that everyone is free to use it and free to redistribute it on certain conditions All of the work on this is done by me in spare time. Special thanks - to Yuri Efimoff <yuri.efimoff@multex.com> for assistance in translation of all this on English. - to bj <bj@ksib.kemerovo.su> for a code samples for windows shortcuts processing. - to Bernd Schemmer for the "REXX Tips and Tricks". - to Vladimir Zabrodsky for the "Album of Algorithms and Techniques for Standard Rexx". - to Guillaume Gay <guillaume.gay@bigfoot.com> for translation of the message file and manual on French. - to Che-ching Wu <ccwu@pie.com.tw> for translation of the message file and manual on Traditional Chinese. - to Bernd Giesen <Bernd.Giesen@t-online.de> for translation of the message file and manual on German. - to Jan-Erik Larka <j-erik.p@telia.com> for translation of the message file and manual on Swedish. - to Alfredo Fern ndez D¡az <alfredo@netropolis-si.com> for translation of the message file and manual on Spanish. -------------------------------------------------- Auto WGet Daemon Great idea and first release (C) 1998 Steve Trubachev Final release (C) 1998-2005 Dmitry A.Steklenev History of changes Version 1.9.5 fix: Fixed the modules initialization in case if BEGINLIBPATH is not empty. Version 1.9.4 fix: Now the daemon works correctly even if it is installed in the directory containing spaces in the name. Version 1.9.3 fix: PPP connection checking for not english-speaking operating systems. fix: Error occured if the event or error log is not defined in the configuration file. fix: Incorrect definition of the event log and error log during the first installation. upd: application/x-msdos-program, audio/x-ms-wma is now supported by plug-ins. upd: The environment variable COMSPEC now is used for determination of a type of a command processor instead of OS2_SHELL variable. Version 1.9.2 upd: Chinese language support. Version 1.9.1 upd: The way of detection of national language is changed. The environment variable "LANG" has the greater priority, than the country code. upd: User name and password now can be entered in the dialog for input a URL. upd: application/x-gzip, audio/mpeg is now supported by plug-ins. upd: application/msword, audio/midi and audio/x-midi mime types are not supported any more by plug-ins. upd: If the plug-in has been called from the mail or news program, the transferred file will be open with the program associated by default for the given type of files. fix: English message file. Thanks to Michael DeBusk <debu4335@earthlink.net> fix: Return to the previous page for embedded audio and video files is not used any more. Version 1.9.0 fix: Extraction of the name of a host from URL. upd: Improved queue planning. add: Netscape and Mozilla plug-in. Version 1.8.3 fix: Spanish language support. Version 1.8.2 fix: Problem with URLs containing '%25'. upd: In wget parameters the key --progress=bar:force is added. add: REDIRECT_TO keyword has been added to additional configuration file. Now you can redirect downloads to another computer. More in detail read "Using in the network" in the manual. Version 1.8.1 upd: Now installation proceeds in the event that is not found any utility of downloading. upd: In wget parameters the key --referer=%h is added. This key allows to download files from some hosts, aggressively reacting on external links. upd: Clipboard monitor are more accurate. fix: SYS3175 in clipboard monitor. Version 1.8.0 fix: SYS3175 on closing the dialog for input a URL via close button. upd: Improved "ToDo" sorting algorithm. upd: Setting DOWNLOADS_ATTEMPTS = 0 disables limitation of attempts of downloading. upd: DOWNLOADS_FROM_SAME_HOST can be changed in additional configuration file. upd: Some words about NFTP are added into manual. add: Clipboard monitor. Version 1.7.2 add: Spanish language support. fix: Now "WP SHARE. SF" and "WP ROOT. SF" files are ignored by the Daemon. upd: The dialog for input a URL is now aligned relative to mouse position. Version 1.7.1 fix: Issues with URLs contain %X symbols, coincided with AWGet symbols. fix: If few URL exist in clipboard, first one will be used. upd: During installation now it's possible to allow/disallow periodical logs pruning. upd: Current directory check up feature during uninstallation has been added. upd: You can point at destination folder when adding URL through object. upd: DOWNLOADS_WINDOW keyword has been added to configuration file. Now you can change download window size and its appearance. Version 1.7.0 fix: Lot of problems with aliases for commands like 'copy' and 'del' in 4OS2. add: In configuration file new key "DOWNLOADS_FROM_SAME_HOST" is added. It allows limit number of downloadable URLs in same time from the same server. Version 1.6.9 add: Swedish language support. upd: The system dialog for input of a URL is replaced with more comfortable dialog. It also troubleshoots additions of a URL containing "%" and ";". fix: Refreshing of a folder after copying the URL object caused hangs on some systems. Version 1.6.8 fix: Prevent possible problems at renaming the object in "Active" folder. fix: Problem with objects containing "%" in the name. fix: Traditional Chinese language support. fix: Download utility selection dialog. Version 1.6.7 add: German language is again supported. Version 1.6.6 add: Traditional Chinese language support. Version 1.6.5 upd: awgadd.cmd now can be called from another REXX program. add: Prune of the log files. See LOG_KEEP in the configuration file. fix: Opening of files in OBJECT REXX required presence of option SHAREREAD. add: Plugins interface. add: After end of downloading URL is placed in the extended attribute .SUBJECT of the downloaded file. URL name and download date store to .COMMENTS. upd: Can't start more than one copy of daemon. add: Utility for stopping down the daemon. add: Added attempts counter for each of URL and maximum number of downloads parameter in the configuration file. add: Now you can create subfolders in ToDo folder with additional configuration files with different parameters. add: Added cURL support (experimental). add: Main configuration file allow to use any another download utility. upd: In case of loosing Internet connection All of active downloads will be closed. add: As the argument for awgadd.cmd you can use filename and path to file, contain URLs. add: You can move active URLs from folder Active to any other or erase. In this case session with this URL will be closed. upd: Many other, I didn't remember all of them. Version 1.5.1.6 add: German language support. add: Chinese manual. upd: The default view for "ToDo" folder is changed on "Tree" because of problems with Netscape Communicator 4.61. Version 1.5.1.5 fix: Value of variable used to store WGet's output is restricted to avoid memory consumption when WGet output is excessive. upd: No above the 20 latest strings of WGet's output are placed to error log. fix: Unnecessary files now deleted with DEL /F even if "SET DELDIR=" is enabled in CONFIG.SYS. upd: Error of WGET, founded at the moment of disconnecting of PPP or SLIP connection at check_connection = 1 are not mistake at all. add: Chinese language support (ChinesesLanguage,Taiwan BIG-5). Version 1.5.1.4 fix: Inquiry of the download directory at installation. fix: Problems with the download directory at use of wget parameters -m (--mirror) or -r (--recursive). upd: The inetwait call is added upd: Processing of shortcuts with URL created with the MSIE or Netscape under windows. Version 1.5.1.2 upd: Because of bugs in old versions OBJECT REXX the subsystem of locking is completely rewritten. upd: The waiting of start of process of downloading with timeout per 30 seconds is added. add: French language support add: Italian language support Version 1.5.1.0 fix: error in installation script. Version 1.5.0.9 fix: error in procedure of check existence of the download directory. Version 1.5.0.8 fix: problem with URLs containing '&()^' Version 1.5.0.7 fix: error in WpsMove(). Version 1.5.0.6 fix: PPP connection check (for PPP1-PPP7) add: At a error the output of the WGet is append in a error log for the further analysis add: uninstallation program. add: new icons :) add: error log file. add: after end of downloading URL is placed in the extended attribute .SUBJECT of the downloaded file. add: Netscape Communicator/2 support. add: after processing URL can move in special folders "Failed" or "Done", depending on what finished downloading. upd: refreshing of folders at moving files. add: object "Add Url" in a folder "Info" for manual addition URL. add: creation of a tray "Auto WGet Daemon" in WarpCenter during installation. fix: many small bugs :) Version 1.4.0.3 fix: problem with URLs containing '%' upd: process complete message automaticaly disposed after one hour