How to Mirror Completely a Website ? (ALL files) in Ultimate Edition 2.3

[ekoice]

The title says pretty all i need to know

Some help is needed here, thank you

[Admin-Amir]

Well Hello ekoice .

How about This one:

Code: Select all

wget --mirror -w 2 -p --html-extension --tries=3 -k -P stackperl.html "http://stackoverflow.com/tags/perl"

There you go with explanation:

I have several web pages on several different sites that I want to mirror completely. This means that I will need images, CSS, etc, and the links need to be converted. This functionality would be similar to using Firefox to "Save Page As" and selecting "Web Page, complete". I'd like to name the files and corresponding directories as something sensible (e.g. myfavpage1.html,myfavpage1.dir).

I do not have access to the servers, and they are not my pages. Here is one sample link: Click Me!

A little more clarification... I have about 100 pages that I want to mirror (many from slow servers), I will be cron'ing the job on Solaris 10 and dumping the results every hour to a samba mount for people to view. And, yes, I have obviously tried wget with several different flags but I haven't gotten the results for which I am looking. So, pointing to the GNU wget page is not really helpful. Let me start with where I am with a simple example.

Mirroring a website on Linux

You almost certainly have wget already. Try wget --help at the command line. If you get an error message, install wget with your Linux distribution's package manager. Or fetch it from the official wget page and compile your own copy from source.

Once you have wget installed correctly, the command line to mirror a website is:

Code: Select all

wget -m -k -K -E http://url/of/web/site

See man wget or wget --help | more for a detailed explanation of each option.

If this command seems to run forever, there may be parts of the site that generate an infinite series of different URLs. You can combat this in many ways, the simplest being to use the -l option to specify how many links "away" from the home page wget should travel. For instance, -l 3 will refuse to download pages more than three clicks away from the home page. You'll have to experiment with different values for -l. Consult man wget for additional workarounds.

Note: some web servers may be set up to "punish" users who download too much, too fast. If you're not careful, using tools like wget could get your IP address banned from the site. You can avoid this problem by using the -w option to specify a delay, in seconds, between page downloads. Usually, this will prevent the web server from viewing your behavior as unacceptable. But your mileage may vary!

If that will not make the job for you let us know please.

[ekoice]

thank you amir

if i want to use this :

Code: Select all

wget -m -k -K -E http://url/of/web/site

with the -l 3 (no more than 3 levels under the top domain), how the command will be ?

thanks again

[Admin-Amir]

Hello

Look if this will give you the answer on the last Question please.

The two types of settings for database mirroring are high safety -synchronous or high performance -asynchronous. The synchronous mode requires the mirror to write a new transaction which will ensure safety but cost in performance. In the asynchronous mode, the session does not wait for the mirror to write which can allow a delay in processing on the mirror. The length of the processing gap depends on how high the work load on the principal database is. A large gap can result on data loss, or less data protection. (MSDN, 2007)

The high safety mode with automatic failover requires a third server instance and is named by Microsoft as the ‘witness’. (MSDN, 2007) The witness verifies that the mirror is connected. A fact worth noting is that the witness and the mirror must be able to communicate even if the primary database is not online. The witness does not serve data, it only watches the transactions between the primary and mirror databases. (MSDN, 2007)

Before a mirror can be implemented, a second instance of the primary database must be established on a different server instance. This can be accomplished by restoring a current backup of the primary database to the second instance WITH NORECOVERY. Any logs that occurred after the backup was made should also be restored WITH NORECOVERY. Both instances should run on the same version of SQL Server, uses full recovery mode, and have equal hardware performance specifications. Also, a windows authentication login must exist on the master of both instances since this will be required for network access. (MSDN, 2007)

After the mirror is established, connect to the primary server instance and at the database node level, open the database properties window, select -configure security- and the mirror wizard launches. The wizard assesses the role of the server instances (primary, mirror, or witness) and provides the server network addresses or endpoint. (Note: Since windows authentication is used here, the endpoint must be configured to use certificates for both the primary and mirror server instances.) There are three safety levels provided by the wizard. They are high security with automatic failover, high security without auto failover, and high performance. To set a high security option, an endpoint for the witness server must be specified first, otherwise, the high security options will not be available. (MSDN, 2007)

As an exercise in writing SQL code, I have taken the following example of how to configure a database mirror in Transact SQL.

1. To use Transact SQL to set up a mirror, first change the recovery mode of the primary database:

USE master;

GO

ALTER DATABASE [primary database]

SET RECOVERY FULL;

GO



2. Create a full backup on the primary server:



BACKUP DATABASE [primary database]

TO DISK = ‘C:\[primary database].bak’

WITH FORMAT

GO



3. Restore the backup on the mirror server:



RESTORE DATABASE [primary database]

FROM DISK = ‘C:\[primary database].bak’

WITH NORECOVERY



***IF THE PATH FOR THE MIRROR IS DIFFERENT, USE THE FOLLOWING MOVE STATEMENT***

MOVE ‘[primary database]_Data’ TO

‘mirror server path’

MOVE ‘[primary database]_Log’ TO

‘mirror server path’

GO



4. Create a log backup:



BACKUP LOG [primary database]

TO DISK = ‘C:\[primary database].bak’

GO

Apply the log backup:

RESTORE LOG [primary database]

FROM DISK = ‘C:\[primary database].bak’

WITH FILE=1, NORECOVERY

GO



Any other logs must be applied in sequence:



RESTORE LOG [primary database]

FROM DISK = ‘C:\[primary database].bak’

WITH FILE=2, NORECOVERY

GO



RESTORE LOG [primary database]

FROM DISK = ‘C:\[primary database].bak’

WITH FILE=3, NORECOVERY

GO



5. Now endpoints must be created on all three server instances. Below is an example of the code for the principal instance:

CREATE ENDPOINT Endpoint_Mirroring

STATE=STARTED

AS TCP (LISTENER_PORT=7022)

FOR DATABASE_MIRRORING (ROLE=PARTNER)

GO

***create a login for the witness server instance***

USE master;

GO

CREATE LOGIN [domain]\witnessuser FROM WINDOWS;

GO

***grant endpoint connection to witness***

GRANT CONNECT ON ENDPOINT::Endpoint_Mirroring TO [domain]\witnessuser;

GO



6. On the mirror server instance, set the primary server as partner:

ALTER DATABASE [mirror database]

SET PARTNER = ‘TCP://[primary database].com:7022’

GO



7. On the principal server instance, set the mirror as the partner:

ALTER DATABASE [primary database]

SET PARTNER = ‘TCP://[mirror database].com:7022’

GO



8. Set the witness on the principal server:

ALTER DATABASE [witness database]

SET WITNESS = ‘TCP://[witness database].com:7022’

GO