Search results
Jul 10, 2020 · A new archive of the entirety of English Wikipedia for Kiwix, an offline browser for web content, is now available for download to anyone who wants to browse offline or have a local backup of the...
Apr 23, 2013 · Having access to nearly all of Wikipedia's articles offline. There are a couple of ways you could do this, and I'll be showing you how to do it with ZIM files via Kiwix (Mac, Windows, Linux), by downloading XML files directly from Wikipedia, and reading XML files with WikiTaxi (Windows).
- Osas Obaiza
Wikipedia themselves support this and there's a variety of tools and torrents available to download compressed version. You can even download the entire dump to a flash drive as long as it's ex-fat format.
- How to self-host a mirror of Wikipedia.org: with Nginx, Kiwix, or MediaWiki/XOWA + Docker
- Intro
- Quickstart
- Getting Started
- Responsible Rehosting Warning
- Table of Contents
- Prerequisites
- Choosing a Wikipedia archive dump
- Method #1: Run a caching proxy in front of Wikipedia.org
- Method #2: Serve the static HTML ZIM archive with Kiwix
Originally published 2019-09-08 on docs.sweeting.me.
The pretty HTML version is here and the source for this guide is on Github.
A summary of how to set up a full Wikipedia.org mirror using three different approaches.
DEMO: https://other-wiki.zervice.io
Unfortunately, Wikipedia attracts lots of hate from people and nation-states who object to certain articles or want to hide information from the public eye.
Wikipedia's infrastructure (2 racks the USA, 1 in Holland, and 1 in Singapore, + CDNs) cant always stand up to large DDoS attacks, but thankfully they provide regular database dumps and static HTML archives to the public, and have permissive licensing that allows for rehosting with modification (even for profit!).
Growing up in China behind the GFC I often experienced Wikipedia unavailability, and in light of the recent DDoS I decided to make a guide for people to help demystify the process of running a mirror. I'm also a big advocate for free access to information, and I'm the maintainer of a major internet archiving project called ArchiveBox (a self-hosted internet archiver powered by headless Chromium).
This aim of this guide is to encourage people to use these publicly available dumps to host Wikipedia mirrors, so that malicious actors don't succeed in limiting public access to one of the world's best sources of information.
A full English Wikipedia.org clone in 3 steps.
DEMO: https://other-wiki.zervice.io
Wikipedia.org itself is powered by a PHP backend called WikiMedia, using MariaDB for data storage, Varnish and Memcached for request and query caching, and ElasticSearch for full-text search. Production Wikipedia.org also runs a number of extra plugins and modules on top of MediaWiki.
🖥 There are several ways to host your own mirror of Wikipedia (with varying complexity):
1.Run a caching proxy in front of Wikipedia.org (disk used on-demand for cache, low CPU use)
2.Serve the static HTML ZIM archive with Kiwix (10~80GB for compressed archive, low CPU use)
3.Run a full MediaWiki server (hardest to set up, ~600GB for XML & database, high CPU use)
💅Don't expect it to look perfect on the first try
⚠️ Be aware that running a publicly-accessible mirror of Wikipedia.org with any kind of framing / content modifications / ads is strongly discouraged. Framing mirrors / proxy mirrors are still a good option for private use, but you need to take additional steps to mirror responsibly if you're setting up a proxy for public use (e.g. robots:noindex, takedown contact info, blocking unlicensed images, etc.).
Luckily, regardless of how you choose to rehost Wikipedia text, you are not breaking any terms and conditions or violating copyright law as long as you don't remove their copyright statements (however, note the article images and videos on Wikimedia.org may not be licensed for re-use).
[TOC]
See the HTML version of this guide for the best browsing experience. See pirate/wikipedia-mirror on Github for example config source, docker-compose files, binaries, folder structure, and more.
1.Provision a server to act as your Wikipedia mirror
You can use a cheap VPS provider like DigitalOcean, Vultr, Hetzner, etc. For the static ZIM archive and MediaWiki server methods you will need significant disk space, so a home server with a cheap external HD may be a better option.
The setup examples below are based on Ubuntu 19.04 running on a home server, however they should work across many other OS's with minimal tweaking (e.g. FreeBSD, macOS, Arch, etc.).
2.Purchase a new domain or create a subdomain to host your mirror
You can use Google Domains, NameCheap, GoDaddy, etc. any registrar will work.
In the setup examples below, replace wiki.example.com with the domain you chose.
a. Running with Nginx
Set the following options in your /opt/wiki/.env config file: UPSTREAM_HOST=wikipedia.org UPSTREAM_WIKI=en.wikipedia.org UPSTREAM_MEDIA=upload.wikimedia.org Then run all the setup steps below under Nginx Reverse Proxy to set up Nginx. Then restart nginx to apply your config with systemctl restart nginx. Your mirror should now be running and proxying requests to Wikipedia.org! Visit https://en.yourdomainhere.com to see it in action (e.g. https://en.wiki.example.com).
b. Running with Caddy
Alternatively, check out a similar setup that uses Caddy instead of Nginx as the reverse proxy: https://github.com/CristianCantoro/wikiproxy
a. Running with Docker
Run kiwix-serve with docker like so: Or create /opt/wiki/docker-compose.yml and run docker-compose up:
b. Running with the static binary
1.Download the latest kiwix-serve binary for your OS & CPU architecture Find the latest release for your architecture here and copy its URL to download it below: https://download.kiwix.org/release/kiwix-tools/ 2.Run kiwix-serve, passing it a port to listen on and your ZIM archive file Your server should now be running! Visit http://en.yourdomainhere.com:8888 to see it in action!
Optional Nginx Reverse Proxy
Set the following options in your /opt/wiki/.env config file: Then run all the setup steps below under Nginx Reverse Proxy to set up Nginx. To run nginx inside docker-compose next to Kiwix, see the Run Nginx via docker-compose section below. Your mirror should now be running and proxying requests to kiwix-serve! Visit https://en.yourdomainhere.com to see it in action (e.g. https://en.wiki.example.com).
Wikipedia is a free content online encyclopedia written and maintained by a community of volunteers, known as Wikipedians, through open collaboration and the wiki software MediaWiki.
BitTorrent, also referred to as simply torrent, is a communication protocol for peer-to-peer file sharing (P2P), which enables users to distribute data and electronic files over the Internet in a decentralized manner. The protocol is developed and maintained by Rainberry, Inc., and was first released in 2001.
People also ask
How do I get the latest version of Wikipedia?
Is Wikipedia a Encyclopedia?
What software does Wikipedia use?
Does Wikipedia have a WYSIWYG visual editor?
The BitTorrent client enables a user to search for and download torrent files using a built-in search box ("Search for torrents") in the main window, which opens the BitTorrent torrent search engine page with the search results in the user's default web browser.
